# Gumbel-Attention for Multi-modal Machine Translation

arXiv: https://arxiv.org/abs/2103.08862 · HTML: https://mlsanity.com/paper/2103.08862/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 29 references that are in the map.
- First posted on arXiv 16 Mar 2021.

## Builds on (most cited first, 12 of 12)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [OpenNMT: Open-source Toolkit for Neural Machine Translation](https://mlsanity.com/paper/1709.03815/) (2017)
- [Multi30K: Multilingual English-German Image Descriptions](https://mlsanity.com/paper/1605.00459/) (2016)
- [Doubly-Attentive Decoder for Multi-modal Neural Machine Translation](https://mlsanity.com/paper/1702.01287/) (2017)
- [Incorporating Global Visual Features into Attention-Based Neural Machine Translation](https://mlsanity.com/paper/1701.06521/) (2017)
- [Distilling Translations with Visual Awareness](https://mlsanity.com/paper/1906.07701/) (2019)
- [A Visual Attention Grounding Neural Model for Multimodal Machine Translation](https://mlsanity.com/paper/1808.08266/) (2018)
- [How Does Selective Mechanism Improve Self-Attention Networks?](https://mlsanity.com/paper/2005.00979/) (2020)
- [LIUM-CVC Submissions for WMT17 Multimodal Translation Task](https://mlsanity.com/paper/1707.04481/) (2017)
- [From Words to Sentences: A Progressive Learning Approach for Zero-resource Machine Translation with Visual Pivots](https://mlsanity.com/paper/1906.00872/) (2019)

## Built on by (most cited first, 20 of 20)

- [MoMa: Efficient Early-Fusion Pre-training with Mixture of Modality-Aware Experts](https://mlsanity.com/paper/2407.21770/) (2024)
- [On Vision Features in Multimodal Machine Translation](https://mlsanity.com/paper/2203.09173/) (2022)
- [VALHALLA: Visual Hallucination for Machine Translation](https://mlsanity.com/paper/2206.00100/) (2022)
- [Neural Machine Translation with Phrase-Level Universal Visual Representations](https://mlsanity.com/paper/2203.10299/) (2022)
- [Tackling Ambiguity with Images: Improved Multimodal Machine Translation and Contrastive Evaluation](https://mlsanity.com/paper/2212.10140/) (2022)
- [Dynamic Tuning Towards Parameter and Inference Efficiency for ViT Adaptation](https://mlsanity.com/paper/2403.11808/) (2024)
- [CLIPTrans: Transferring Visual Knowledge with Pre-trained Models for Multimodal Machine Translation](https://mlsanity.com/paper/2308.15226/) (2023)
- [Hausa Visual Genome: A Dataset for Multi-Modal English to Hausa Machine Translation](https://mlsanity.com/paper/2205.01133/) (2022)
- [Distill the Image to Nowhere: Inversion Knowledge Distillation for Multimodal Machine Translation](https://mlsanity.com/paper/2210.04468/) (2022)
- [Increasing Visual Awareness in Multimodal Neural Machine Translation from an Information Theoretic Perspective](https://mlsanity.com/paper/2210.08478/) (2022)
- [Decoupling Weighing and Selecting for Integrating Multiple Graph Pre-training Tasks](https://mlsanity.com/paper/2403.01400/) (2024)
- [Pay Better Attention to Attention: Head Selection in Multilingual and Multi-Domain Sequence Modeling](https://mlsanity.com/paper/2106.10840/) (2021)
- [Low-resource Neural Machine Translation with Cross-modal Alignment](https://mlsanity.com/paper/2210.06716/) (2022)
- [Towards Zero-Shot Multimodal Machine Translation](https://mlsanity.com/paper/2407.13579/) (2024)
- [Iterative Adversarial Attack on Image-guided Story Ending Generation](https://mlsanity.com/paper/2305.13208/) (2023)
- [Dual-branch Prompting for Multimodal Machine Translation](https://mlsanity.com/paper/2507.17588/) (2025)
- [ConECT Dataset: Overcoming Data Scarcity in Context-Aware E-Commerce MT](https://mlsanity.com/paper/2506.04929/) (2025)
- [GIIFT: Graph-guided Inductive Image-free Multimodal Machine Translation](https://mlsanity.com/paper/2507.18562/) (2025)
- [Empowering Denoising Sequential Recommendation with Large Language Model Embeddings](https://mlsanity.com/paper/2510.04239/) (2025)
- [Improving Visual Sensitivity of LLMs on Multimodal Machine Translation with Metric-based Loss Weighting](https://mlsanity.com/paper/2609.31169/) (2026)

## Abstract (arXiv)

Multi-modal machine translation (MMT) improves translation quality by introducing visual information. However, the existing MMT model ignores the problem that the image will bring information irrelevant to the text, causing much noise to the model and affecting the translation quality. This paper proposes a novel Gumbel-Attention for multi-modal machine translation, which selects the text-related parts of the image features. Specifically, different from the previous attention-based method, we first use a differentiable method to select the image information and automatically remove the useless parts of the image features. Experiments prove that our method retains the image features related to the text, and the remaining parts help the MMT model generates better translations.