# Towards Zero-shot Cross-lingual Image Retrieval and Tagging

arXiv: https://arxiv.org/abs/2109.07622 · HTML: https://mlsanity.com/paper/2109.07622/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 41 references that are in the map.
- First posted on arXiv 15 Sep 2021.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Word Translation Without Parallel Data](https://mlsanity.com/paper/1710.04087/) (2017)
- [Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond](https://mlsanity.com/paper/1812.10464/) (2018)
- [Multi30K: Multilingual English-German Image Descriptions](https://mlsanity.com/paper/1605.00459/) (2016)
- [Multilingual Universal Sentence Encoder for Semantic Retrieval](https://mlsanity.com/paper/1907.04307/) (2019)
- [COCO-CN for Cross-Lingual Image Tagging, Captioning and Retrieval](https://mlsanity.com/paper/1805.08661/) (2018)
- [Findings of the Second Shared Task on Multimodal Machine Translation and Multilingual Image Description](https://mlsanity.com/paper/1710.07177/) (2017)
- [AI Challenger : A Large-scale Dataset for Going Deeper in Image Understanding](https://mlsanity.com/paper/1711.06475/) (2017)
- [STAIR Captions: Constructing a Large-Scale Japanese Image Caption Dataset](https://mlsanity.com/paper/1705.00823/) (2017)
- [Learning Cross-Lingual Sentence Representations via a Multi-task Dual-Encoder Model](https://mlsanity.com/paper/1810.12836/) (2018)
- [Visual Search at Alibaba](https://mlsanity.com/paper/2102.04674/) (2021)
- [Visual Search at eBay](https://mlsanity.com/paper/1706.03154/) (2017)
- [Towards Zero-shot Cross-lingual Image Retrieval](https://mlsanity.com/paper/2012.05107/) (2020)
- [MULE: Multimodal Universal Language Embedding](https://mlsanity.com/paper/1909.03493/) (2019)
- [Learning to Scale Multilingual Representations for Vision-Language Tasks](https://mlsanity.com/paper/2004.04312/) (2020)
- [Visual Search at Pinterest](https://mlsanity.com/paper/1505.07647/) (2015)
- [Bridge Correlational Neural Networks for Multilingual Multimodal Representation Learning](https://mlsanity.com/paper/1510.03519/) (2015)
- [Image Pivoting for Learning Multilingual Multimodal Representations](https://mlsanity.com/paper/1707.07601/) (2017)
- [Deep Learning based Large Scale Visual Recommendation and Search for E-Commerce](https://mlsanity.com/paper/1703.02344/) (2017)
- [Multi-Head Attention with Diversity for Learning Grounded Multilingual Multimodal Representations](https://mlsanity.com/paper/1910.00058/) (2019)
- [Zero-resource Machine Translation by Multimodal Encoder-decoder Network with Multimedia Pivot](https://mlsanity.com/paper/1611.04503/) (2016)
- [Lessons learned in multilingual grounded language learning](https://mlsanity.com/paper/1809.07615/) (2018)
- [Image search using multilingual texts: a cross-modal learning approach between image and text](https://mlsanity.com/paper/1903.11299/) (2019)
- [Aligning Multilingual Word Embeddings for Cross-Modal Retrieval Task](https://mlsanity.com/paper/1910.03291/) (2019)
- [Multilingual Multi-modal Embeddings for Natural Language Processing](https://mlsanity.com/paper/1702.01101/) (2017)
- [Multitask Text-to-Visual Embedding with Titles and Clickthrough Data](https://mlsanity.com/paper/1905.13339/) (2019)

## Built on by (most cited first, 4 of 4)

- [IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages](https://mlsanity.com/paper/2201.11732/) (2022)
- [M-SpeechCLIP: Leveraging Large-Scale, Pre-Trained Models for Multilingual Speech to Image Retrieval](https://mlsanity.com/paper/2211.01180/) (2022)
- [C-CLIP: Contrastive Image-Text Encoders to Close the Descriptive-Commentative Gap](https://mlsanity.com/paper/2309.03921/) (2023)
- [Embedded Heterogeneous Attention Transformer for Cross-lingual Image Captioning](https://mlsanity.com/paper/2307.09915/) (2023)

## Abstract (arXiv)

There has been a recent spike in interest in multi-modal Language and Vision problems. On the language side, most of these models primarily focus on English since most multi-modal datasets are monolingual. We try to bridge this gap with a zero-shot approach for learning multi-modal representations using cross-lingual pre-training on the text side. We present a simple yet practical approach for building a cross-lingual image retrieval model which trains on a monolingual training dataset but can be used in a zero-shot cross-lingual fashion during inference. We also introduce a new objective function which tightens the text embedding clusters by pushing dissimilar texts away from each other. For evaluation, we introduce a new 1K multi-lingual MSCOCO2014 caption test dataset (XTD10) in 7 languages that we collected using a crowdsourcing platform. We use this as the test set for zero-shot model performance across languages. We also demonstrate how a cross-lingual model can be used for downstream tasks like multi-lingual image tagging in a zero shot manner. XTD10 dataset is made publicly available here: https://github.com/adobe-research/Cross-lingual-Test-Dataset-XTD10.