# Aligning Multilingual Word Embeddings for Cross-Modal Retrieval Task

arXiv: https://arxiv.org/abs/1910.03291 · HTML: https://mlsanity.com/paper/1910.03291/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Oct 2019.

## Built on by (most cited first, 6 of 6)

- [RQUGE: Reference-Free Metric for Evaluating Question Generation by Answering the Question](https://mlsanity.com/paper/2211.01482/) (2022)
- [Syntax-Aware Graph-to-Graph Transformer for Semantic Role Labelling](https://mlsanity.com/paper/2104.07704/) (2021)
- [Towards Zero-shot Cross-lingual Image Retrieval and Tagging](https://mlsanity.com/paper/2109.07622/) (2021)
- [Multilingual Multimodality: A Taxonomical Survey of Datasets, Techniques, Challenges and Opportunities](https://mlsanity.com/paper/2210.16960/) (2022)
- [Visual Grounding of Inter-lingual Word-Embeddings](https://mlsanity.com/paper/2209.03714/) (2022)
- [Discriminating Form and Meaning in Multilingual Models with Minimal-Pair ABX Tasks](https://mlsanity.com/paper/2505.17747/) (2025)

## Abstract (arXiv)

In this paper, we propose a new approach to learn multimodal multilingual embeddings for matching images and their relevant captions in two languages. We combine two existing objective functions to make images and captions close in a joint embedding space while adapting the alignment of word embeddings between existing languages in our model. We show that our approach enables better generalization, achieving state-of-the-art performance in text-to-image and image-to-text retrieval task, and caption-caption similarity task. Two multimodal multilingual datasets are used for evaluation: Multi30k with German and English captions and Microsoft-COCO with English and Japanese captions.