# A Comprehensive Survey on Cross-modal Retrieval

arXiv: https://arxiv.org/abs/1607.06215 · HTML: https://mlsanity.com/paper/1607.06215/

## Key facts

- Cited by 62 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Jul 2016.

## Built on by (most cited first, 30 of 62)

- [GME: Improving Universal Multimodal Retrieval by Multimodal LLMs](https://mlsanity.com/paper/2412.16855/) (2024)
- [VLP: A Survey on Vision-Language Pre-training](https://mlsanity.com/paper/2202.09061/) (2022)
- [Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learning](https://mlsanity.com/paper/2212.04994/) (2022)
- [Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2412.02104/) (2024)
- [A survey of multimodal deep generative models](https://mlsanity.com/paper/2207.02127/) (2022)
- [Product1M: Towards Weakly Supervised Instance-Level Product Retrieval via Cross-modal Pretraining](https://mlsanity.com/paper/2107.14572/) (2021)
- [One Perturbation is Enough: On Generating Universal Adversarial Perturbations against Vision-Language Pre-training Models](https://mlsanity.com/paper/2406.05491/) (2024)
- [A Survey of Large Language Model Empowered Agents for Recommendation and Search: Towards Next-Generation Information Retrieval](https://mlsanity.com/paper/2503.05659/) (2025)
- [Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrieval](https://mlsanity.com/paper/2505.19650/) (2025)
- [From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunities](https://mlsanity.com/paper/2311.00308/) (2023)
- [Vision+X: A Survey on Multimodal Learning in the Light of Data](https://mlsanity.com/paper/2210.02884/) (2022)
- [What do you MEME? Generating Explanations for Visual Semantic Role Labelling in Memes](https://mlsanity.com/paper/2212.00715/) (2022)
- [Learning Structural Representations for Recipe Generation and Food Retrieval](https://mlsanity.com/paper/2110.01209/) (2021)
- [Semi-Supervised Semantic Segmentation Based on Pseudo-Labels: A Survey](https://mlsanity.com/paper/2403.01909/) (2024)
- [RoarGraph: A Projected Bipartite Graph for Efficient Cross-Modal Approximate Nearest Neighbor Search](https://mlsanity.com/paper/2408.08933/) (2024)
- [VaLiD: Mitigating the Hallucination of Large Vision Language Models by Visual Layer Fusion Contrastive Decoding](https://mlsanity.com/paper/2411.15839/) (2024)
- [MEMEX: Detecting Explanatory Evidence for Memes via Knowledge-Enriched Contextualization](https://mlsanity.com/paper/2305.15913/) (2023)
- [Multimodal Lecture Presentations Dataset: Understanding Multimodality in Educational Slides](https://mlsanity.com/paper/2208.08080/) (2022)
- [Knowledge-aware Text-Image Retrieval for Remote Sensing Images](https://mlsanity.com/paper/2405.03373/) (2024)
- [BadCM: Invisible Backdoor Attack Against Cross-Modal Learning](https://mlsanity.com/paper/2410.02182/) (2024)
- [MMMORRF: Multimodal Multilingual Modularized Reciprocal Rank Fusion](https://mlsanity.com/paper/2503.20698/) (2025)
- [ReID5o: Achieving Omni Multi-modal Person Re-identification in a Single Model](https://mlsanity.com/paper/2506.09385/) (2025)
- [Multimodal Neural Databases](https://mlsanity.com/paper/2305.01447/) (2023)
- [ProbVLM: Probabilistic Adapter for Frozen Vision-Language Models](https://mlsanity.com/paper/2307.00398/) (2023)
- [Synaptic Plasticity Models and Bio-Inspired Unsupervised Deep Learning: A Survey](https://mlsanity.com/paper/2307.16236/) (2023)
- [Integrating Information Theory and Adversarial Learning for Cross-modal Retrieval](https://mlsanity.com/paper/2104.04991/) (2021)
- [Person Retrieval in Surveillance Using Textual Query: A Review](https://mlsanity.com/paper/2105.02414/) (2021)
- [Improving Video Retrieval by Adaptive Margin](https://mlsanity.com/paper/2303.05093/) (2023)
- [Achieving Human Parity on Visual Question Answering](https://mlsanity.com/paper/2111.08896/) (2021)
- [SwAMP: Swapped Assignment of Multi-Modal Pairs for Cross-Modal Retrieval](https://mlsanity.com/paper/2111.05814/) (2021)

## Abstract (arXiv)

In recent years, cross-modal retrieval has drawn much attention due to the rapid growth of multimodal data. It takes one type of data as the query to retrieve relevant data of another type. For example, a user can use a text to retrieve relevant pictures or videos. Since the query and its retrieved results can be of different modalities, how to measure the content similarity between different modalities of data remains a challenge. Various methods have been proposed to deal with such a problem. In this paper, we first review a number of representative methods for cross-modal retrieval and classify them into two main groups: 1) real-valued representation learning, and 2) binary representation learning. Real-valued representation learning methods aim to learn real-valued common representations for different modalities of data. To speed up the cross-modal retrieval, a number of binary representation learning methods are proposed to map different modalities of data into a common Hamming space. Then, we introduce several multimodal datasets in the community, and show the experimental results on two commonly used multimodal datasets. The comparison reveals the characteristic of different kinds of cross-modal retrieval methods, which is expected to benefit both practical applications and future research. Finally, we discuss open problems and future research directions.