# Instance-aware Image and Sentence Matching with Selective Multimodal LSTM

arXiv: https://arxiv.org/abs/1611.05588 · HTML: https://mlsanity.com/paper/1611.05588/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Nov 2016.

## Built on by (most cited first, 30 of 43)

- [Probabilistic Embeddings for Cross-Modal Retrieval](https://mlsanity.com/paper/2101.05068/) (2021)
- [Vision-Language Pre-training: Basics, Recent Advances, and Future Trends](https://mlsanity.com/paper/2210.09263/) (2022)
- [Are Gender-Neutral Queries Really Gender-Neutral? Mitigating Gender Bias in Image Search](https://mlsanity.com/paper/2109.05433/) (2021)
- [UPop: Unified and Progressive Pruning for Compressing Vision-Language Transformers](https://mlsanity.com/paper/2301.13741/) (2023)
- [CrossGET: Cross-Guided Ensemble of Tokens for Accelerating Vision-Language Transformers](https://mlsanity.com/paper/2305.17455/) (2023)
- [VT-CLIP: Enhancing Vision-Language Models with Visual-guided Texts](https://mlsanity.com/paper/2112.02399/) (2021)
- [Vision-Language Intelligence: Tasks, Representation Learning, and Large Models](https://mlsanity.com/paper/2203.01922/) (2022)
- [Cross-Lingual Cross-Modal Retrieval with Noise-Robust Learning](https://mlsanity.com/paper/2208.12526/) (2022)
- [VisualSparta: An Embarrassingly Simple Approach to Large-scale Text-to-Image Search with Weighted Bag-of-words](https://mlsanity.com/paper/2101.00265/) (2021)
- [ICECAP: Information Concentrated Entity-aware Image Captioning](https://mlsanity.com/paper/2108.02050/) (2021)
- [Plug-and-Play Regulators for Image-Text Matching](https://mlsanity.com/paper/2303.13371/) (2023)
- [ELIP: Enhanced Visual-Language Foundation Models for Image Retrieval](https://mlsanity.com/paper/2502.15682/) (2025)
- [Discrete-continuous Action Space Policy Gradient-based Attention for Image-Text Matching](https://mlsanity.com/paper/2104.10406/) (2021)
- [CLIP2GAN: Towards Bridging Text with the Latent Space of GANs](https://mlsanity.com/paper/2211.15045/) (2022)
- [Towards Fast and Accurate Image-Text Retrieval with Self-Supervised Fine-Grained Alignment](https://mlsanity.com/paper/2308.14009/) (2023)
- [DocumentCLIP: Linking Figures and Main Body Text in Reflowed Documents](https://mlsanity.com/paper/2306.06306/) (2023)
- [Integrating Information Theory and Adversarial Learning for Cross-modal Retrieval](https://mlsanity.com/paper/2104.04991/) (2021)
- [More Than Just Attention: Improving Cross-Modal Attentions with Contrastive Constraints for Image-Text Matching](https://mlsanity.com/paper/2105.09597/) (2021)
- [LILE: Look In-Depth before Looking Elsewhere -- A Dual Attention Network using Transformers for Cross-Modal Information Retrieval in Histopathology Archives](https://mlsanity.com/paper/2203.01445/) (2022)
- [Object-Centric Open-Vocabulary Image-Retrieval with Aggregated Features](https://mlsanity.com/paper/2309.14999/) (2023)
- [A Comprehensive Empirical Study of Vision-Language Pre-trained Model for Supervised Cross-Modal Retrieval](https://mlsanity.com/paper/2201.02772/) (2022)
- [IIITD-20K: Dense captioning for Text-Image ReID](https://mlsanity.com/paper/2305.04497/) (2023)
- [GSSF: Generalized Structural Sparse Function for Deep Cross-modal Metric Learning](https://mlsanity.com/paper/2410.15266/) (2024)
- [Object-Aware Query Perturbation for Cross-Modal Image-Text Retrieval](https://mlsanity.com/paper/2407.12346/) (2024)
- [GraspMamba: A Mamba-based Language-driven Grasp Detection Framework with Hierarchical Feature Learning](https://mlsanity.com/paper/2409.14403/) (2024)
- [An Unsupervised Sampling Approach for Image-Sentence Matching Using Document-Level Structural Information](https://mlsanity.com/paper/2104.02605/) (2021)
- [Uncertainty-based Cross-Modal Retrieval with Probabilistic Representations](https://mlsanity.com/paper/2204.09268/) (2022)
- [FOR: Finetuning for Object Level Open Vocabulary Image Retrieval](https://mlsanity.com/paper/2412.18806/) (2024)
- [Memory Enhanced Embedding Learning for Cross-Modal Video-Text Retrieval](https://mlsanity.com/paper/2103.15686/) (2021)
- [Learning to embed semantic similarity for joint image-text retrieval](https://mlsanity.com/paper/2210.03838/) (2022)

## Abstract (arXiv)

Effective image and sentence matching depends on how to well measure their global visual-semantic similarity. Based on the observation that such a global similarity arises from a complex aggregation of multiple local similarities between pairwise instances of image (objects) and sentence (words), we propose a selective multimodal Long Short-Term Memory network (sm-LSTM) for instance-aware image and sentence matching. The sm-LSTM includes a multimodal context-modulated attention scheme at each timestep that can selectively attend to a pair of instances of image and sentence, by predicting pairwise instance-aware saliency maps for image and sentence. For selected pairwise instances, their representations are obtained based on the predicted saliency maps, and then compared to measure their local similarity. By similarly measuring multiple local similarities within a few timesteps, the sm-LSTM sequentially aggregates them with hidden states to obtain a final matching score as the desired global similarity. Extensive experiments show that our model can well match image and sentence with complex content, and achieve the state-of-the-art results on two public benchmark datasets.