# An empirical study on the effectiveness of images in Multimodal Neural Machine Translation

arXiv: https://arxiv.org/abs/1707.00995 · HTML: https://mlsanity.com/paper/1707.00995/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Jul 2017.

## Built on by (most cited first, 10 of 10)

- [On Vision Features in Multimodal Machine Translation](https://mlsanity.com/paper/2203.09173/) (2022)
- [Neural Machine Translation with Phrase-Level Universal Visual Representations](https://mlsanity.com/paper/2203.10299/) (2022)
- [Product-oriented Machine Translation with Cross-modal Cross-lingual Pre-training](https://mlsanity.com/paper/2108.11119/) (2021)
- [Increasing Visual Awareness in Multimodal Neural Machine Translation from an Information Theoretic Perspective](https://mlsanity.com/paper/2210.08478/) (2022)
- [IndicVisionBench: Benchmarking Cultural and Multilingual Understanding in VLMs](https://mlsanity.com/paper/2511.04727/) (2025)
- [Chitranuvad: Adapting Multi-Lingual LLMs for Multimodal Translation](https://mlsanity.com/paper/2502.20420/) (2025)
- [Low-resource Neural Machine Translation with Cross-modal Alignment](https://mlsanity.com/paper/2210.06716/) (2022)
- [Deep soccer captioning with transformer: dataset, semantics-related losses, and multi-level evaluation](https://mlsanity.com/paper/2202.05728/) (2022)
- [3AM: An Ambiguity-Aware Multi-Modal Machine Translation Dataset](https://mlsanity.com/paper/2404.18413/) (2024)
- [Incorporating Probing Signals into Multimodal Machine Translation via Visual Question-Answering Pairs](https://mlsanity.com/paper/2310.17133/) (2023)

## Abstract (arXiv)

In state-of-the-art Neural Machine Translation (NMT), an attention mechanism is used during decoding to enhance the translation. At every step, the decoder uses this mechanism to focus on different parts of the source sentence to gather the most useful information before outputting its target word. Recently, the effectiveness of the attention mechanism has also been explored for multimodal tasks, where it becomes possible to focus both on sentence parts and image regions that they describe. In this paper, we compare several attention mechanism on the multimodal translation task (English, image to German) and evaluate the ability of the model to make use of images to improve translation. We surpass state-of-the-art scores on the Multi30k data set, we nevertheless identify and report different misbehavior of the machine while translating.