# Transformer Reasoning Network for Image-Text Matching and Retrieval

arXiv: https://arxiv.org/abs/2004.09144 · HTML: https://mlsanity.com/paper/2004.09144/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Apr 2020.

## Built on by (most cited first, 19 of 19)

- [Combining EfficientNet and Vision Transformers for Video Deepfake Detection](https://mlsanity.com/paper/2107.02612/) (2021)
- [Text2Pos: Text-to-Point-Cloud Cross-Modal Localization](https://mlsanity.com/paper/2203.15125/) (2022)
- [Cross-Modal Retrieval Augmentation for Multi-Modal Classification](https://mlsanity.com/paper/2104.08108/) (2021)
- [Is CLIP the main roadblock for fine-grained open-world perception?](https://mlsanity.com/paper/2404.03539/) (2024)
- [Telling the What while Pointing to the Where: Multimodal Queries for Image Retrieval](https://mlsanity.com/paper/2102.04980/) (2021)
- [Text-to-Motion Retrieval: Towards Joint Understanding of Human Motion Data and Natural Language](https://mlsanity.com/paper/2305.15842/) (2023)
- [Recurrent Vision Transformer for Solving Visual Reasoning Problems](https://mlsanity.com/paper/2111.14576/) (2021)
- [ALADIN: Distilling Fine-grained Alignment Scores for Efficient Image-Text Matching and Retrieval](https://mlsanity.com/paper/2207.14757/) (2022)
- [Towards Efficient Cross-Modal Visual Textual Retrieval using Transformer-Encoder Deep Features](https://mlsanity.com/paper/2106.00358/) (2021)
- [A Spatio-Temporal Attentive Network for Video-Based Crowd Counting](https://mlsanity.com/paper/2208.11339/) (2022)
- [Variable Assignment Invariant Neural Networks for Learning Logic Programs](https://mlsanity.com/paper/2408.10709/) (2024)
- [Do Lessons from Metric Learning Generalize to Image-Caption Retrieval?](https://mlsanity.com/paper/2202.07474/) (2022)
- [Reducing Predictive Feature Suppression in Resource-Constrained Contrastive Image-Caption Retrieval](https://mlsanity.com/paper/2204.13382/) (2022)
- [A review of machine learning approaches, challenges and prospects for computational tumor pathology](https://mlsanity.com/paper/2206.01728/) (2022)
- [Transformer-Based Multi-modal Proposal and Re-Rank for Wikipedia Image-Caption Matching](https://mlsanity.com/paper/2206.10436/) (2022)
- [MLP: Motion Label Prior for Temporal Sentence Localization in Untrimmed 3D Human Motions](https://mlsanity.com/paper/2404.13657/) (2024)
- [Modeling Temporal-Modal Entity Graph for Procedural Multimodal Machine Comprehension](https://mlsanity.com/paper/2204.02566/) (2022)
- [Progressive Feature Mining and External Knowledge-Assisted Text-Pedestrian Image Retrieval](https://mlsanity.com/paper/2308.11994/) (2023)
- [HNC: Leveraging Hard Negative Captions towards Models with Fine-Grained Visual-Linguistic Comprehension Capabilities](https://mlsanity.com/paper/2605.06157/) (2026)

## Abstract (arXiv)

Image-text matching is an interesting and fascinating task in modern AI research. Despite the evolution of deep-learning-based image and text processing systems, multi-modal matching remains a challenging problem. In this work, we consider the problem of accurate image-text matching for the task of multi-modal large-scale information retrieval. State-of-the-art results in image-text matching are achieved by inter-playing image and text features from the two different processing pipelines, usually using mutual attention mechanisms. However, this invalidates any chance to extract separate visual and textual features needed for later indexing steps in large-scale retrieval systems. In this regard, we introduce the Transformer Encoder Reasoning Network (TERN), an architecture built upon one of the modern relationship-aware self-attentive architectures, the Transformer Encoder (TE). This architecture is able to separately reason on the two different modalities and to enforce a final common abstract concept space by sharing the weights of the deeper transformer layers. Thanks to this design, the implemented network is able to produce compact and very rich visual and textual features available for the successive indexing step. Experiments are conducted on the MS-COCO dataset, and we evaluate the results using a discounted cumulative gain metric with relevance computed exploiting caption similarities, in order to assess possibly non-exact but relevant search results. We demonstrate that on this metric we are able to achieve state-of-the-art results in the image retrieval task. Our code is freely available at https://github.com/mesnico/TERN.