# Scene Graph Based Fusion Network For Image-Text Retrieval

arXiv: https://arxiv.org/abs/2303.11090 · HTML: https://mlsanity.com/paper/2303.11090/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 25 references that are in the map.
- First posted on arXiv 20 Mar 2023.

## Builds on (most cited first, 17 of 17)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015)
- [CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification](https://mlsanity.com/paper/2103.14899/) (2021)
- [Stacked Cross Attention for Image-Text Matching](https://mlsanity.com/paper/1803.08024/) (2018)
- [Attention Bottlenecks for Multimodal Fusion](https://mlsanity.com/paper/2107.00135/) (2021)
- [DeepViT: Towards Deeper Vision Transformer](https://mlsanity.com/paper/2103.11886/) (2021)
- [Similarity Reasoning and Filtration for Image-Text Matching](https://mlsanity.com/paper/2101.01368/) (2021)
- [Learning Two-Branch Neural Networks for Image-Text Matching Tasks](https://mlsanity.com/paper/1704.03470/) (2017)
- [RelTR: Relation Transformer for Scene Graph Generation](https://mlsanity.com/paper/2201.11460/) (2022)
- [Graph Structured Network for Image-Text Matching](https://mlsanity.com/paper/2004.00277/) (2020)
- [Cross-modal Scene Graph Matching for Relationship-aware Image-Text Retrieval](https://mlsanity.com/paper/1910.05134/) (2019)
- [Position Focused Attention Network for Image-Text Matching](https://mlsanity.com/paper/1907.09748/) (2019)
- [Vision-Language Intelligence: Tasks, Representation Learning, and Large Models](https://mlsanity.com/paper/2203.01922/) (2022)
- [Step-Wise Hierarchical Alignment Network for Image-Text Matching](https://mlsanity.com/paper/2106.06509/) (2021)
- [Unsupervised Vision-Language Parsing: Seamlessly Bridging Visual Scene Graphs with Language Structures via Dependency Relationships](https://mlsanity.com/paper/2203.14260/) (2022)
- [A Deep Local and Global Scene-Graph Matching for Image-Text Retrieval](https://mlsanity.com/paper/2106.02400/) (2021)

## Built on by (most cited first, 2 of 2)

- [SCENIR: Visual Semantic Clarity through Unsupervised Scene Graph Retrieval](https://mlsanity.com/paper/2505.15867/) (2025)
- [Knowledge-Guided Manipulation Using Multi-Task Reinforcement Learning](https://mlsanity.com/paper/2603.24083/) (2026)

## Abstract (arXiv)

A critical challenge to image-text retrieval is how to learn accurate correspondences between images and texts. Most existing methods mainly focus on coarse-grained correspondences based on co-occurrences of semantic objects, while failing to distinguish the fine-grained local correspondences. In this paper, we propose a novel Scene Graph based Fusion Network (dubbed SGFN), which enhances the images'/texts' features through intra- and cross-modal fusion for image-text retrieval. To be specific, we design an intra-modal hierarchical attention fusion to incorporate semantic contexts, such as objects, attributes, and relationships, into images'/texts' feature vectors via scene graphs, and a cross-modal attention fusion to combine the contextual semantics and local fusion via contextual vectors. Extensive experiments on public datasets Flickr30K and MSCOCO show that our SGFN performs better than quite a few SOTA image-text retrieval methods.