# Saliency-Guided Attention Network for Image-Sentence Matching

arXiv: https://arxiv.org/abs/1904.09471 · HTML: https://mlsanity.com/paper/1904.09471/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Apr 2019.

## Built on by (most cited first, 21 of 21)

- [Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videos](https://mlsanity.com/paper/2109.03223/) (2021)
- [Similarity Reasoning and Filtration for Image-Text Matching](https://mlsanity.com/paper/2101.01368/) (2021)
- [Exploring a Fine-Grained Multiscale Method for Cross-Modal Remote Sensing Image Retrieval](https://mlsanity.com/paper/2204.09868/) (2022)
- [Image-text Retrieval: A Survey on Recent Research and Development](https://mlsanity.com/paper/2203.14713/) (2022)
- [Disentangled High Quality Salient Object Detection](https://mlsanity.com/paper/2108.03551/) (2021)
- [Cross-modal Semantic Enhanced Interaction for Image-Sentence Retrieval](https://mlsanity.com/paper/2210.08908/) (2022)
- [Direction-Oriented Visual-semantic Embedding Model for Remote Sensing Image-text Retrieval](https://mlsanity.com/paper/2310.08276/) (2023)
- [USER: Unified Semantic Enhancement with Momentum Contrast for Image-Text Retrieval](https://mlsanity.com/paper/2301.06844/) (2023)
- [Ask&Confirm: Active Detail Enriching for Cross-Modal Retrieval with Partial Query](https://mlsanity.com/paper/2103.01654/) (2021)
- [ELIP: Enhanced Visual-Language Foundation Models for Image Retrieval](https://mlsanity.com/paper/2502.15682/) (2025)
- [Grounded Image Text Matching with Mismatched Relation Reasoning](https://mlsanity.com/paper/2308.01236/) (2023)
- [3SHNet: Boosting Image-Sentence Retrieval via Visual Semantic-Spatial Self-Highlighting](https://mlsanity.com/paper/2404.17273/) (2024)
- [LILE: Look In-Depth before Looking Elsewhere -- A Dual Attention Network using Transformers for Cross-Modal Information Retrieval in Histopathology Archives](https://mlsanity.com/paper/2203.01445/) (2022)
- [Object-Centric Open-Vocabulary Image-Retrieval with Aggregated Features](https://mlsanity.com/paper/2309.14999/) (2023)
- [Constructing Phrase-level Semantic Labels to Form Multi-Grained Supervision for Image-Text Retrieval](https://mlsanity.com/paper/2109.05523/) (2021)
- [Hierarchical Matching and Reasoning for Multi-Query Image Retrieval](https://mlsanity.com/paper/2306.14460/) (2023)
- [Object-Aware Query Perturbation for Cross-Modal Image-Text Retrieval](https://mlsanity.com/paper/2407.12346/) (2024)
- [Negative Sample is Negative in Its Own Way: Tailoring Negative Sentences for Image-Text Retrieval](https://mlsanity.com/paper/2111.03349/) (2021)
- [Two-stream Hierarchical Similarity Reasoning for Image-text Matching](https://mlsanity.com/paper/2203.05349/) (2022)
- [FOR: Finetuning for Object Level Open Vocabulary Image Retrieval](https://mlsanity.com/paper/2412.18806/) (2024)
- [Hire: Hybrid-modal Interaction with Multiple Relational Enhancements for Image-Text Matching](https://mlsanity.com/paper/2406.18579/) (2024)

## Abstract (arXiv)

This paper studies the task of matching image and sentence, where learning appropriate representations across the multi-modal data appears to be the main challenge. Unlike previous approaches that predominantly deploy symmetrical architecture to represent both modalities, we propose Saliency-guided Attention Network (SAN) that asymmetrically employs visual and textual attention modules to learn the fine-grained correlation intertwined between vision and language. The proposed SAN mainly includes three components: saliency detector, Saliency-weighted Visual Attention (SVA) module, and Saliency-guided Textual Attention (STA) module. Concretely, the saliency detector provides the visual saliency information as the guidance for the two attention modules. SVA is designed to leverage the advantage of the saliency information to improve discrimination of visual representations. By fusing the visual information from SVA and textual information as a multi-modal guidance, STA learns discriminative textual representations that are highly sensitive to visual clues. Extensive experiments demonstrate SAN can substantially improve the state-of-the-art results on the benchmark Flickr30K and MSCOCO datasets by a large margin.