# A Visual Attention Grounding Neural Model for Multimodal Machine Translation

arXiv: https://arxiv.org/abs/1808.08266 · HTML: https://mlsanity.com/paper/1808.08266/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Aug 2018.

## Built on by (most cited first, 22 of 22)

- [WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning](https://mlsanity.com/paper/2103.01913/) (2021)
- [UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-training](https://mlsanity.com/paper/2104.00332/) (2021)
- [Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translation](https://mlsanity.com/paper/2105.14462/) (2021)
- [Grounding 'Grounding' in NLP](https://mlsanity.com/paper/2106.02192/) (2021)
- [VALHALLA: Visual Hallucination for Machine Translation](https://mlsanity.com/paper/2206.00100/) (2022)
- [Neural Machine Translation with Phrase-Level Universal Visual Representations](https://mlsanity.com/paper/2203.10299/) (2022)
- [LVP-M3: Language-aware Visual Prompt for Multilingual Multimodal Machine Translation](https://mlsanity.com/paper/2210.15461/) (2022)
- [Gumbel-Attention for Multi-modal Machine Translation](https://mlsanity.com/paper/2103.08862/) (2021)
- [A Survey on Low-Resource Neural Machine Translation](https://mlsanity.com/paper/2107.04239/) (2021)
- [CLIPTrans: Transferring Visual Knowledge with Pre-trained Models for Multimodal Machine Translation](https://mlsanity.com/paper/2308.15226/) (2023)
- [AOMD: An Analogy-aware Approach to Offensive Meme Detection on Social Media](https://mlsanity.com/paper/2106.11229/) (2021)
- [Distill the Image to Nowhere: Inversion Knowledge Distillation for Multimodal Machine Translation](https://mlsanity.com/paper/2210.04468/) (2022)
- [Product-oriented Machine Translation with Cross-modal Cross-lingual Pre-training](https://mlsanity.com/paper/2108.11119/) (2021)
- [A Survey on Multi-modal Machine Translation: Tasks, Methods and Challenges](https://mlsanity.com/paper/2405.12669/) (2024)
- [Low-resource Neural Machine Translation with Cross-modal Alignment](https://mlsanity.com/paper/2210.06716/) (2022)
- [Detecting Concrete Visual Tokens for Multimodal Machine Translation](https://mlsanity.com/paper/2403.03075/) (2024)
- [Supervised Visual Attention for Simultaneous Multimodal Machine Translation](https://mlsanity.com/paper/2201.09324/) (2022)
- [Impact of Visual Context on Noisy Multimodal NMT: An Empirical Study for English to Indian Languages](https://mlsanity.com/paper/2308.16075/) (2023)
- [Multilingual Multimodality: A Taxonomical Survey of Datasets, Techniques, Challenges and Opportunities](https://mlsanity.com/paper/2210.16960/) (2022)
- [Backretrieval: An Image-Pivoted Evaluation Metric for Cross-Lingual Text Representations Without Parallel Corpora](https://mlsanity.com/paper/2105.04971/) (2021)
- [Utilizing Language-Image Pretraining for Efficient and Robust Bilingual Word Alignment](https://mlsanity.com/paper/2205.11616/) (2022)
- [Enhancing E-commerce Product Title Translation with Retrieval-Augmented Generation and Large Language Models](https://mlsanity.com/paper/2409.12880/) (2024)

## Abstract (arXiv)

We introduce a novel multimodal machine translation model that utilizes parallel visual and textual information. Our model jointly optimizes the learning of a shared visual-language embedding and a translator. The model leverages a visual attention grounding mechanism that links the visual semantics with the corresponding textual semantics. Our approach achieves competitive state-of-the-art results on the Multi30K and the Ambiguous COCO datasets. We also collected a new multilingual multimodal product description dataset to simulate a real-world international online shopping scenario. On this dataset, our visual attention grounding model outperforms other methods by a large margin.