# Modality-specific Cross-modal Similarity Measurement with Recurrent Attention Network

arXiv: https://arxiv.org/abs/1708.04776 · HTML: https://mlsanity.com/paper/1708.04776/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Aug 2017.

## Built on by (most cited first, 11 of 11)

- [AdvCLIP: Downstream-agnostic Adversarial Examples in Multimodal Contrastive Learning](https://mlsanity.com/paper/2308.07026/) (2023)
- [The ObjectFolder Benchmark: Multisensory Learning with Neural and Real Objects](https://mlsanity.com/paper/2306.00956/) (2023)
- [Adversarial Attacks of Vision Tasks in the Past 10 Years: A Survey](https://mlsanity.com/paper/2410.23687/) (2024)
- [Dynamic Enhancement Network for Partial Multi-modality Person Re-identification](https://mlsanity.com/paper/2305.15762/) (2023)
- [Alternative Telescopic Displacement: An Efficient Multimodal Alignment Method](https://mlsanity.com/paper/2306.16950/) (2023)
- [Enhancing Visible-Infrared Person Re-identification with Modality- and Instance-aware Visual Prompt Learning](https://mlsanity.com/paper/2406.12316/) (2024)
- [Zoom and Shift are All You Need](https://mlsanity.com/paper/2406.08866/) (2024)
- [Rebalanced Vision-Language Retrieval Considering Structure-Aware Distillation](https://mlsanity.com/paper/2412.10761/) (2024)
- [A Comprehensive Empirical Study of Vision-Language Pre-trained Model for Supervised Cross-Modal Retrieval](https://mlsanity.com/paper/2201.02772/) (2022)
- [Music-to-Text Synaesthesia: Generating Descriptive Text from Music Recordings](https://mlsanity.com/paper/2210.00434/) (2022)
- [A Unified Optimal Transport Framework for Cross-Modal Retrieval with Noisy Labels](https://mlsanity.com/paper/2403.13480/) (2024)

## Abstract (arXiv)

Nowadays, cross-modal retrieval plays an indispensable role to flexibly find information across different modalities of data. Effectively measuring the similarity between different modalities of data is the key of cross-modal retrieval. Different modalities such as image and text have imbalanced and complementary relationships, which contain unequal amount of information when describing the same semantics. For example, images often contain more details that cannot be demonstrated by textual descriptions and vice versa. Existing works based on Deep Neural Network (DNN) mostly construct one common space for different modalities to find the latent alignments between them, which lose their exclusive modality-specific characteristics. Different from the existing works, we propose modality-specific cross-modal similarity measurement (MCSM) approach by constructing independent semantic space for each modality, which adopts end-to-end framework to directly generate modality-specific cross-modal similarity without explicit common representation. For each semantic space, modality-specific characteristics within one modality are fully exploited by recurrent attention network, while the data of another modality is projected into this space with attention based joint embedding to utilize the learned attention weights for guiding the fine-grained cross-modal correlation learning, which can capture the imbalanced and complementary relationships between different modalities. Finally, the complementarity between the semantic spaces for different modalities is explored by adaptive fusion of the modality-specific cross-modal similarities to perform cross-modal retrieval. Experiments on the widely-used Wikipedia and Pascal Sentence datasets as well as our constructed large-scale XMediaNet dataset verify the effectiveness of our proposed approach, outperforming 9 state-of-the-art methods.