# IMRAM: Iterative Matching with Recurrent Attention Memory for Cross-Modal Image-Text Retrieval

arXiv: https://arxiv.org/abs/2003.03772 · HTML: https://mlsanity.com/paper/2003.03772/

## Key facts

- Cited by 115 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Mar 2020.

## Built on by (most cited first, 30 of 115)

- [Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval](https://mlsanity.com/paper/2104.00650/) (2021)
- [VLP: A Survey on Vision-Language Pre-training](https://mlsanity.com/paper/2202.09061/) (2022)
- [Similarity Reasoning and Filtration for Image-Text Matching](https://mlsanity.com/paper/2101.01368/) (2021)
- [Towards artificial general intelligence via a multimodal foundation model](https://mlsanity.com/paper/2110.14378/) (2021)
- [Semantically Self-Aligned Network for Text-to-Image Part-aware Person Re-identification](https://mlsanity.com/paper/2107.12666/) (2021)
- [Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models](https://mlsanity.com/paper/2307.14061/) (2023)
- [VLMbench: A Compositional Benchmark for Vision-and-Language Manipulation](https://mlsanity.com/paper/2206.08522/) (2022)
- [Image-text Retrieval: A Survey on Recent Research and Development](https://mlsanity.com/paper/2203.14713/) (2022)
- [Disentangling Visual Embeddings for Attributes and Objects](https://mlsanity.com/paper/2205.08536/) (2022)
- [Boosting Transferability in Vision-Language Attacks via Diversification along the Intersection Region of Adversarial Trajectory](https://mlsanity.com/paper/2403.12445/) (2024)
- [Improving Cross-Modal Retrieval with Set of Diverse Embeddings](https://mlsanity.com/paper/2211.16761/) (2022)
- [ViSTA: Vision and Scene Text Aggregation for Cross-Modal Retrieval](https://mlsanity.com/paper/2203.16778/) (2022)
- [Generative Cross-Modal Retrieval: Memorizing Images in Multimodal Language Models for Retrieval and Beyond](https://mlsanity.com/paper/2402.10805/) (2024)
- [BiCro: Noisy Correspondence Rectification for Multi-modality Data via Bi-directional Cross-modal Similarity Consistency](https://mlsanity.com/paper/2303.12419/) (2023)
- [Step-Wise Hierarchical Alignment Network for Image-Text Matching](https://mlsanity.com/paper/2106.06509/) (2021)
- [Noisy Correspondence Learning with Meta Similarity Correction](https://mlsanity.com/paper/2304.06275/) (2023)
- [Contrastive Vision-Language Pre-training with Limited Resources](https://mlsanity.com/paper/2112.09331/) (2021)
- [TENT: Connect Language Models with IoT Sensors for Zero-Shot Activity Recognition](https://mlsanity.com/paper/2311.08245/) (2023)
- [Structured Multi-modal Feature Embedding and Alignment for Image-Sentence Retrieval](https://mlsanity.com/paper/2108.02417/) (2021)
- [Open Vocabulary Object Detection with Proposal Mining and Prediction Equalization](https://mlsanity.com/paper/2206.11134/) (2022)
- [Cross-modal Active Complementary Learning with Self-refining Correspondence](https://mlsanity.com/paper/2310.17468/) (2023)
- [Iterative Shrinking for Referring Expression Grounding Using Deep Reinforcement Learning](https://mlsanity.com/paper/2103.05187/) (2021)
- [Semantic-Aligned Adversarial Evolution Triangle for High-Transferability Vision-Language Attack](https://mlsanity.com/paper/2411.02669/) (2024)
- [Zero-Shot Compositional Concept Learning](https://mlsanity.com/paper/2107.05176/) (2021)
- [LSNet: See Large, Focus Small](https://mlsanity.com/paper/2503.23135/) (2025)
- [Cross-modal Semantic Enhanced Interaction for Image-Sentence Retrieval](https://mlsanity.com/paper/2210.08908/) (2022)
- [Fully and Weakly Supervised Referring Expression Segmentation with End-to-End Learning](https://mlsanity.com/paper/2212.10278/) (2022)
- [Is An Image Worth Five Sentences? A New Look into Semantics for Image-Text Matching](https://mlsanity.com/paper/2110.02623/) (2021)
- [CODER: Coupled Diversity-Sensitive Momentum Contrastive Learning for Image-Text Retrieval](https://mlsanity.com/paper/2208.09843/) (2022)
- [USER: Unified Semantic Enhancement with Momentum Contrast for Image-Text Retrieval](https://mlsanity.com/paper/2301.06844/) (2023)

## Abstract (arXiv)

Enabling bi-directional retrieval of images and texts is important for understanding the correspondence between vision and language. Existing methods leverage the attention mechanism to explore such correspondence in a fine-grained manner. However, most of them consider all semantics equally and thus align them uniformly, regardless of their diverse complexities. In fact, semantics are diverse (i.e. involving different kinds of semantic concepts), and humans usually follow a latent structure to combine them into understandable languages. It may be difficult to optimally capture such sophisticated correspondences in existing methods. In this paper, to address such a deficiency, we propose an Iterative Matching with Recurrent Attention Memory (IMRAM) method, in which correspondences between images and texts are captured with multiple steps of alignments. Specifically, we introduce an iterative matching scheme to explore such fine-grained correspondence progressively. A memory distillation unit is used to refine alignment knowledge from early steps to later ones. Experiment results on three benchmark datasets, i.e. Flickr8K, Flickr30K, and MS COCO, show that our IMRAM achieves state-of-the-art performance, well demonstrating its effectiveness. Experiments on a practical business advertisement dataset, named Ads, further validates the applicability of our method in practical scenarios.