# AMC: Attention guided Multi-modal Correlation Learning for Image Search

arXiv: https://arxiv.org/abs/1704.00763 · HTML: https://mlsanity.com/paper/1704.00763/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Apr 2017.

## Built on by (most cited first, 8 of 8)

- [Modality-aware Mutual Learning for Multi-modal Medical Image Segmentation](https://mlsanity.com/paper/2107.09842/) (2021)
- [Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applications](https://mlsanity.com/paper/2302.00389/) (2023)
- [Dynamic Enhancement Network for Partial Multi-modality Person Re-identification](https://mlsanity.com/paper/2305.15762/) (2023)
- [Proposal-free One-stage Referring Expression via Grid-Word Cross-Attention](https://mlsanity.com/paper/2105.02061/) (2021)
- [A Survey on Image-text Multimodal Models](https://mlsanity.com/paper/2309.15857/) (2023)
- [SLAN: Self-Locator Aided Network for Cross-Modal Understanding](https://mlsanity.com/paper/2211.16208/) (2022)
- [TextAug: Test time Text Augmentation for Multimodal Person Re-identification](https://mlsanity.com/paper/2312.01605/) (2023)
- [Weakly-Supervised Action Detection Guided by Audio Narration](https://mlsanity.com/paper/2205.05895/) (2022)

## Abstract (arXiv)

Given a user's query, traditional image search systems rank images according to its relevance to a single modality (e.g., image content or surrounding text). Nowadays, an increasing number of images on the Internet are available with associated meta data in rich modalities (e.g., titles, keywords, tags, etc.), which can be exploited for better similarity measure with queries. In this paper, we leverage visual and textual modalities for image search by learning their correlation with input query. According to the intent of query, attention mechanism can be introduced to adaptively balance the importance of different modalities. We propose a novel Attention guided Multi-modal Correlation (AMC) learning method which consists of a jointly learned hierarchy of intra and inter-attention networks. Conditioned on query's intent, intra-attention networks (i.e., visual intra-attention network and language intra-attention network) attend on informative parts within each modality; a multi-modal inter-attention network promotes the importance of the most query-relevant modalities. In experiments, we evaluate AMC models on the search logs from two real world image search engines and show a significant boost on the ranking of user-clicked images in search results. Additionally, we extend AMC models to caption ranking task on COCO dataset and achieve competitive results compared with recent state-of-the-arts.