# Multi-Head Attention with Diversity for Learning Grounded Multilingual Multimodal Representations

arXiv: https://arxiv.org/abs/1910.00058 · HTML: https://mlsanity.com/paper/1910.00058/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Sep 2019.

## Built on by (most cited first, 10 of 10)

- [IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages](https://mlsanity.com/paper/2201.11732/) (2022)
- [Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models](https://mlsanity.com/paper/2103.08849/) (2021)
- [Scene Graph as Pivoting: Inference-time Image-free Unsupervised Multimodal Machine Translation with Visual Scene Hallucination](https://mlsanity.com/paper/2305.12256/) (2023)
- [LookHere: Vision Transformers with Directed Attention Generalize and Extrapolate](https://mlsanity.com/paper/2405.13985/) (2024)
- [Towards Zero-shot Cross-lingual Image Retrieval and Tagging](https://mlsanity.com/paper/2109.07622/) (2021)
- [Manifestations of Xenophobia in AI Systems](https://mlsanity.com/paper/2212.07877/) (2022)
- [A survey of diversity quantification in natural language processing: The why, what, where and how](https://mlsanity.com/paper/2507.20858/) (2025)
- [Multilingual Multimodality: A Taxonomical Survey of Datasets, Techniques, Challenges and Opportunities](https://mlsanity.com/paper/2210.16960/) (2022)
- [Patch Pruning Strategy Based on Robust Statistical Measures of Attention Weight Diversity in Vision Transformers](https://mlsanity.com/paper/2507.19175/) (2025)
- [Entropy and Attention Dynamics in Small Language Models: A Trace-Level Structural Analysis on the TruthfulQA Benchmark](https://mlsanity.com/paper/2604.03589/) (2026)

## Abstract (arXiv)

With the aim of promoting and understanding the multilingual version of image search, we leverage visual object detection and propose a model with diverse multi-head attention to learn grounded multilingual multimodal representations. Specifically, our model attends to different types of textual semantics in two languages and visual objects for fine-grained alignments between sentences and images. We introduce a new objective function which explicitly encourages attention diversity to learn an improved visual-semantic embedding space. We evaluate our model in the German-Image and English-Image matching tasks on the Multi30K dataset, and in the Semantic Textual Similarity task with the English descriptions of visual content. Results show that our model yields a significant performance gain over other methods in all of the three tasks.