# Multimodal Compact Bilinear Pooling for Multimodal Neural Machine Translation

arXiv: https://arxiv.org/abs/1703.08084 · HTML: https://mlsanity.com/paper/1703.08084/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Mar 2017.

## Built on by (most cited first, 8 of 8)

- [Neural Machine Translation with Phrase-Level Universal Visual Representations](https://mlsanity.com/paper/2203.10299/) (2022)
- [A Survey on Multi-modal Machine Translation: Tasks, Methods and Challenges](https://mlsanity.com/paper/2405.12669/) (2024)
- [Low-resource Neural Machine Translation with Cross-modal Alignment](https://mlsanity.com/paper/2210.06716/) (2022)
- [Deep soccer captioning with transformer: dataset, semantics-related losses, and multi-level evaluation](https://mlsanity.com/paper/2202.05728/) (2022)
- [Multimodal Neural Machine Translation with Search Engine Based Image Retrieval](https://mlsanity.com/paper/2208.00767/) (2022)
- [Quality-Aware Multimodal Biometric Recognition](https://mlsanity.com/paper/2112.05827/) (2021)
- [Exploring the Necessity of Visual Modality in Multimodal Machine Translation using Authentic Datasets](https://mlsanity.com/paper/2404.06107/) (2024)
- [Generalization algorithm of multimodal pre-training model based on graph-text self-supervised training](https://mlsanity.com/paper/2302.10315/) (2023)

## Abstract (arXiv)

In state-of-the-art Neural Machine Translation, an attention mechanism is used during decoding to enhance the translation. At every step, the decoder uses this mechanism to focus on different parts of the source sentence to gather the most useful information before outputting its target word. Recently, the effectiveness of the attention mechanism has also been explored for multimodal tasks, where it becomes possible to focus both on sentence parts and image regions. Approaches to pool two modalities usually include element-wise product, sum or concatenation. In this paper, we evaluate the more advanced Multimodal Compact Bilinear pooling method, which takes the outer product of two vectors to combine the attention features for the two modalities. This has been previously investigated for visual question answering. We try out this approach for multimodal image caption translation and show improvements compared to basic combination methods.