# Expressing Objects just like Words: Recurrent Visual Embedding for Image-Text Matching

arXiv: https://arxiv.org/abs/2002.08510 · HTML: https://mlsanity.com/paper/2002.08510/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Feb 2020.

## Built on by (most cited first, 15 of 15)

- [Similarity Reasoning and Filtration for Image-Text Matching](https://mlsanity.com/paper/2101.01368/) (2021)
- [From Multi-View to Hollow-3D: Hallucinated Hollow-3D R-CNN for 3D Object Detection](https://mlsanity.com/paper/2107.14391/) (2021)
- [Structured Multi-modal Feature Embedding and Alignment for Image-Sentence Retrieval](https://mlsanity.com/paper/2108.02417/) (2021)
- [Parts2Words: Learning Joint Embedding of Point Clouds and Texts by Bidirectional Matching between Parts and Words](https://mlsanity.com/paper/2107.01872/) (2021)
- [Plug-and-Play Regulators for Image-Text Matching](https://mlsanity.com/paper/2303.13371/) (2023)
- [Grounded Image Text Matching with Mismatched Relation Reasoning](https://mlsanity.com/paper/2308.01236/) (2023)
- [Unifying Two-Stream Encoders with Transformers for Cross-Modal Retrieval](https://mlsanity.com/paper/2308.04343/) (2023)
- [HGAN: Hierarchical Graph Alignment Network for Image-Text Retrieval](https://mlsanity.com/paper/2212.08281/) (2022)
- [Beat: Bi-directional One-to-Many Embedding Alignment for Text-based Person Retrieval](https://mlsanity.com/paper/2406.05620/) (2024)
- [Constructing Phrase-level Semantic Labels to Form Multi-Grained Supervision for Image-Text Retrieval](https://mlsanity.com/paper/2109.05523/) (2021)
- [Understanding Attention for Vision-and-Language Tasks](https://mlsanity.com/paper/2208.08104/) (2022)
- [Hierarchical Matching and Reasoning for Multi-Query Image Retrieval](https://mlsanity.com/paper/2306.14460/) (2023)
- [Two-stream Hierarchical Similarity Reasoning for Image-text Matching](https://mlsanity.com/paper/2203.05349/) (2022)
- [Unified Loss of Pair Similarity Optimization for Vision-Language Retrieval](https://mlsanity.com/paper/2209.13869/) (2022)
- [T-EMDE: Sketching-based global similarity for cross-modal retrieval](https://mlsanity.com/paper/2105.04242/) (2021)

## Abstract (arXiv)

Existing image-text matching approaches typically infer the similarity of an image-text pair by capturing and aggregating the affinities between the text and each independent object of the image. However, they ignore the connections between the objects that are semantically related. These objects may collectively determine whether the image corresponds to a text or not. To address this problem, we propose a Dual Path Recurrent Neural Network (DP-RNN) which processes images and sentences symmetrically by recurrent neural networks (RNN). In particular, given an input image-text pair, our model reorders the image objects based on the positions of their most related words in the text. In the same way as extracting the hidden features from word embeddings, the model leverages RNN to extract high-level object features from the reordered object inputs. We validate that the high-level object features contain useful joint information of semantically related objects, which benefit the retrieval task. To compute the image-text similarity, we incorporate a Multi-attention Cross Matching Model into DP-RNN. It aggregates the affinity between objects and words with cross-modality guided attention and self-attention. Our model achieves the state-of-the-art performance on Flickr30K dataset and competitive performance on MS-COCO dataset. Extensive experiments demonstrate the effectiveness of our model.