# Distilling Translations with Visual Awareness

arXiv: https://arxiv.org/abs/1906.07701 · HTML: https://mlsanity.com/paper/1906.07701/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Jun 2019.

## Built on by (most cited first, 30 of 30)

- [Gender Bias in Machine Translation](https://mlsanity.com/paper/2104.06001/) (2021)
- [Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translation](https://mlsanity.com/paper/2105.14462/) (2021)
- [Vision-Language Intelligence: Tasks, Representation Learning, and Large Models](https://mlsanity.com/paper/2203.01922/) (2022)
- [VALHALLA: Visual Hallucination for Machine Translation](https://mlsanity.com/paper/2206.00100/) (2022)
- [Neural Machine Translation with Phrase-Level Universal Visual Representations](https://mlsanity.com/paper/2203.10299/) (2022)
- [Tackling Ambiguity with Images: Improved Multimodal Machine Translation and Contrastive Evaluation](https://mlsanity.com/paper/2212.10140/) (2022)
- [Exploring Better Text Image Translation with Multimodal Codebook](https://mlsanity.com/paper/2305.17415/) (2023)
- [Migician: Revealing the Magic of Free-Form Multi-Image Grounding in Multimodal Large Language Models](https://mlsanity.com/paper/2501.05767/) (2025)
- [Vision Matters When It Should: Sanity Checking Multimodal Machine Translation Models](https://mlsanity.com/paper/2109.03415/) (2021)
- [LVP-M3: Language-aware Visual Prompt for Multilingual Multimodal Machine Translation](https://mlsanity.com/paper/2210.15461/) (2022)
- [Gumbel-Attention for Multi-modal Machine Translation](https://mlsanity.com/paper/2103.08862/) (2021)
- [MSCTD: A Multimodal Sentiment Chat Translation Dataset](https://mlsanity.com/paper/2202.13645/) (2022)
- [Universal Multimodal Representation for Language Understanding](https://mlsanity.com/paper/2301.03344/) (2023)
- [Distill the Image to Nowhere: Inversion Knowledge Distillation for Multimodal Machine Translation](https://mlsanity.com/paper/2210.04468/) (2022)
- [Product-oriented Machine Translation with Cross-modal Cross-lingual Pre-training](https://mlsanity.com/paper/2108.11119/) (2021)
- [Increasing Visual Awareness in Multimodal Neural Machine Translation from an Information Theoretic Perspective](https://mlsanity.com/paper/2210.08478/) (2022)
- [Exploiting Multimodal Reinforcement Learning for Simultaneous Machine Translation](https://mlsanity.com/paper/2102.11387/) (2021)
- [M2C: Towards Automatic Multimodal Manga Complement](https://mlsanity.com/paper/2310.17130/) (2023)
- [A Survey on Multi-modal Machine Translation: Tasks, Methods and Challenges](https://mlsanity.com/paper/2405.12669/) (2024)
- [Low-resource Neural Machine Translation with Cross-modal Alignment](https://mlsanity.com/paper/2210.06716/) (2022)
- [Multimodal Neural Machine Translation with Search Engine Based Image Retrieval](https://mlsanity.com/paper/2208.00767/) (2022)
- [Adding Multimodal Capabilities to a Text-only Translation Model](https://mlsanity.com/paper/2403.03045/) (2024)
- [Detecting Concrete Visual Tokens for Multimodal Machine Translation](https://mlsanity.com/paper/2403.03075/) (2024)
- [Visual Cues and Error Correction for Translation Robustness](https://mlsanity.com/paper/2103.07352/) (2021)
- [Transfer Learning for Sequence Generation: from Single-source to Multi-source](https://mlsanity.com/paper/2105.14809/) (2021)
- [Impact of Visual Context on Noisy Multimodal NMT: An Empirical Study for English to Indian Languages](https://mlsanity.com/paper/2308.16075/) (2023)
- [Dual-branch Prompting for Multimodal Machine Translation](https://mlsanity.com/paper/2507.17588/) (2025)
- [Few-shot Domain-Adaptive Visually-fused Event Detection from Text](https://mlsanity.com/paper/2305.03517/) (2023)
- [Exploring the Necessity of Visual Modality in Multimodal Machine Translation using Authentic Datasets](https://mlsanity.com/paper/2404.06107/) (2024)
- [GIIFT: Graph-guided Inductive Image-free Multimodal Machine Translation](https://mlsanity.com/paper/2507.18562/) (2025)

## Abstract (arXiv)

Previous work on multimodal machine translation has shown that visual information is only needed in very specific cases, for example in the presence of ambiguous words where the textual context is not sufficient. As a consequence, models tend to learn to ignore this information. We propose a translate-and-refine approach to this problem where images are only used by a second stage decoder. This approach is trained jointly to generate a good first draft translation and to improve over this draft by (i) making better use of the target language textual context (both left and right-side contexts) and (ii) making use of visual context. This approach leads to the state of the art results. Additionally, we show that it has the ability to recover from erroneous or missing words in the source language.