# Image Pivoting for Learning Multilingual Multimodal Representations

arXiv: https://arxiv.org/abs/1707.07601 · HTML: https://mlsanity.com/paper/1707.07601/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Jul 2017.

## Built on by (most cited first, 17 of 17)

- [Wukong: A 100 Million Large-scale Chinese Cross-modal Pre-training Benchmark](https://mlsanity.com/paper/2202.06767/) (2022)
- [Decoupling the Role of Data, Attention, and Losses in Multimodal Transformers](https://mlsanity.com/paper/2102.00529/) (2021)
- [UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-training](https://mlsanity.com/paper/2104.00332/) (2021)
- [IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages](https://mlsanity.com/paper/2201.11732/) (2022)
- [MURAL: Multimodal, Multitask Retrieval Across Languages](https://mlsanity.com/paper/2109.05125/) (2021)
- [Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models](https://mlsanity.com/paper/2103.08849/) (2021)
- [Retrieve Fast, Rerank Smart: Cooperative and Joint Approaches for Improved Cross-Modal Retrieval](https://mlsanity.com/paper/2103.11920/) (2021)
- [Crossing the Conversational Chasm: A Primer on Natural Language Processing for Multilingual Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2104.08570/) (2021)
- [Multilingual Multimodal Learning with Machine Translated Text](https://mlsanity.com/paper/2210.13134/) (2022)
- [Babel-ImageNet: Massively Multilingual Evaluation of Vision-and-Language Representations](https://mlsanity.com/paper/2306.08658/) (2023)
- [Towards Zero-shot Cross-lingual Image Retrieval and Tagging](https://mlsanity.com/paper/2109.07622/) (2021)
- [Generalizing Multimodal Pre-training into Multilingual via Language Acquisition](https://mlsanity.com/paper/2206.11091/) (2022)
- [MuMUR : Multilingual Multimodal Universal Retrieval](https://mlsanity.com/paper/2208.11553/) (2022)
- [Meta-learning For Vision-and-language Cross-lingual Transfer](https://mlsanity.com/paper/2305.14843/) (2023)
- [Multilingual Multimodality: A Taxonomical Survey of Datasets, Techniques, Challenges and Opportunities](https://mlsanity.com/paper/2210.16960/) (2022)
- [Accessible Instruction-Following Agent](https://mlsanity.com/paper/2305.06358/) (2023)
- [TransMeme: A Multi-Agent Framework for Cross-Cultural Meme Transcreation](https://mlsanity.com/paper/2608.27127/) (2026)

## Abstract (arXiv)

In this paper we propose a model to learn multimodal multilingual representations for matching images and sentences in different languages, with the aim of advancing multilingual versions of image search and image understanding. Our model learns a common representation for images and their descriptions in two different languages (which need not be parallel) by considering the image as a pivot between two languages. We introduce a new pairwise ranking loss function which can handle both symmetric and asymmetric similarity between the two modalities. We evaluate our models on image-description ranking for German and English, and on semantic textual similarity of image descriptions in English. In both cases we achieve state-of-the-art performance.