# Generative Imagination Elevates Machine Translation

arXiv: https://arxiv.org/abs/2009.09654 · HTML: https://mlsanity.com/paper/2009.09654/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Sep 2020.

## Built on by (most cited first, 22 of 22)

- [Visualize Before You Write: Imagination-Guided Open-Ended Text Generation](https://mlsanity.com/paper/2210.03765/) (2022)
- [VALHALLA: Visual Hallucination for Machine Translation](https://mlsanity.com/paper/2206.00100/) (2022)
- [Neural Machine Translation with Phrase-Level Universal Visual Representations](https://mlsanity.com/paper/2203.10299/) (2022)
- [Exploring Better Text Image Translation with Multimodal Codebook](https://mlsanity.com/paper/2305.17415/) (2023)
- [Imagination-Augmented Natural Language Understanding](https://mlsanity.com/paper/2204.08535/) (2022)
- [CLIPTrans: Transferring Visual Knowledge with Pre-trained Models for Multimodal Machine Translation](https://mlsanity.com/paper/2308.15226/) (2023)
- [Hausa Visual Genome: A Dataset for Multi-Modal English to Hausa Machine Translation](https://mlsanity.com/paper/2205.01133/) (2022)
- [Distill the Image to Nowhere: Inversion Knowledge Distillation for Multimodal Machine Translation](https://mlsanity.com/paper/2210.04468/) (2022)
- [A Multi-Modal Context Reasoning Approach for Conditional Inference on Joint Textual and Visual Clues](https://mlsanity.com/paper/2305.04530/) (2023)
- [Training Multimedia Event Extraction With Generated Images and Captions](https://mlsanity.com/paper/2306.08966/) (2023)
- [Learning to Imagine: Visually-Augmented Natural Language Generation](https://mlsanity.com/paper/2305.16944/) (2023)
- [Bridging the Gap between Synthetic and Authentic Images for Multimodal Machine Translation](https://mlsanity.com/paper/2310.13361/) (2023)
- [A Survey on Multi-modal Machine Translation: Tasks, Methods and Challenges](https://mlsanity.com/paper/2405.12669/) (2024)
- [Vision Enhancing LLMs: Empowering Multimodal Knowledge Storage and Sharing in LLMs](https://mlsanity.com/paper/2311.15759/) (2023)
- [A Multimodal In-Context Tuning Approach for E-Commerce Product Description Generation](https://mlsanity.com/paper/2402.13587/) (2024)
- [Detecting Euphemisms with Literal Descriptions and Visual Imagery](https://mlsanity.com/paper/2211.04576/) (2022)
- [Training on Synthetic Data Beats Real Data in Multimodal Relation Extraction](https://mlsanity.com/paper/2312.03025/) (2023)
- [Dual-branch Prompting for Multimodal Machine Translation](https://mlsanity.com/paper/2507.17588/) (2025)
- [Movie Box Office Prediction With Self-Supervised and Visually Grounded Pretraining](https://mlsanity.com/paper/2304.10311/) (2023)
- [Paint Outside the Box: Synthesizing and Selecting Training Data for Visual Grounding](https://mlsanity.com/paper/2412.00684/) (2024)
- [GIIFT: Graph-guided Inductive Image-free Multimodal Machine Translation](https://mlsanity.com/paper/2507.18562/) (2025)
- [Improving Visual Sensitivity of LLMs on Multimodal Machine Translation with Metric-based Loss Weighting](https://mlsanity.com/paper/2609.31169/) (2026)

## Abstract (arXiv)

There are common semantics shared across text and images. Given a sentence in a source language, whether depicting the visual scene helps translation into a target language? Existing multimodal neural machine translation methods (MNMT) require triplets of bilingual sentence - image for training and tuples of source sentence - image for inference. In this paper, we propose ImagiT, a novel machine translation method via visual imagination. ImagiT first learns to generate visual representation from the source sentence, and then utilizes both source sentence and the "imagined representation" to produce a target translation. Unlike previous methods, it only needs the source sentence at the inference time. Experiments demonstrate that ImagiT benefits from visual imagination and significantly outperforms the text-only neural machine translation baselines. Further analysis reveals that the imagination process in ImagiT helps fill in missing information when performing the degradation strategy.