# Image-to-Image Retrieval by Learning Similarity between Scene Graphs

arXiv: https://arxiv.org/abs/2012.14700 · HTML: https://mlsanity.com/paper/2012.14700/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Dec 2020.

## Built on by (most cited first, 23 of 23)

- [Relational Context Learning for Human-Object Interaction Detection](https://mlsanity.com/paper/2304.04997/) (2023)
- [Dysen-VDM: Empowering Dynamics-aware Text-to-Video Diffusion with LLMs](https://mlsanity.com/paper/2308.13812/) (2023)
- [HiKER-SGG: Hierarchical Knowledge Enhanced Robust Scene Graph Generation](https://mlsanity.com/paper/2403.12033/) (2024)
- [Locality-Aware Zero-Shot Human-Object Interaction Detection](https://mlsanity.com/paper/2505.19503/) (2025)
- [Graph Neural Networks in Vision-Language Image Understanding: A Survey](https://mlsanity.com/paper/2303.03761/) (2023)
- [Statistical Confidence Rescoring for Robust 3D Scene Graph Generation from Multi-View Images](https://mlsanity.com/paper/2508.06546/) (2025)
- [Automated Floodwater Depth Estimation Using Large Multimodal Model for Rapid Flood Mapping](https://mlsanity.com/paper/2402.16684/) (2024)
- [SPAN: Learning Similarity between Scene Graphs and Images with Transformers](https://mlsanity.com/paper/2304.00590/) (2023)
- [Generalized Unbiased Scene Graph Generation](https://mlsanity.com/paper/2308.04802/) (2023)
- [Frequency-based Matcher for Long-tailed Semantic Segmentation](https://mlsanity.com/paper/2406.03917/) (2024)
- [SATURN: Autoregressive Image Generation Guided by Scene Graphs](https://mlsanity.com/paper/2508.14502/) (2025)
- [SPA-VAE: Similar-Parts-Assignment for Unsupervised 3D Point Cloud Generation](https://mlsanity.com/paper/2203.07825/) (2022)
- [SCENIR: Visual Semantic Clarity through Unsupervised Scene Graph Retrieval](https://mlsanity.com/paper/2505.15867/) (2025)
- [Response-G1: Explicit Scene Graph Modeling for Proactive Streaming Video Understanding](https://mlsanity.com/paper/2605.07575/) (2026)
- [Interactive Visual Pattern Search on Graph Data via Graph Representation Learning](https://mlsanity.com/paper/2202.09459/) (2022)
- [VITR: Augmenting Vision Transformers with Relation-Focused Learning for Cross-Modal Information Retrieval](https://mlsanity.com/paper/2302.06350/) (2023)
- [Training Free Graph Neural Networks for Graph Matching](https://mlsanity.com/paper/2201.05349/) (2022)
- [Image Similarity using An Ensemble of Context-Sensitive Models](https://mlsanity.com/paper/2401.07951/) (2024)
- [Multi-Modal Scene Graph with Kolmogorov-Arnold Experts for Audio-Visual Question Answering](https://mlsanity.com/paper/2511.23304/) (2025)
- [Through the PRISm: Importance-Aware Scene Graphs for Image Retrieval](https://mlsanity.com/paper/2512.18407/) (2025)
- [VENUS: Visual Editing with Noise Inversion Using Scene Graphs](https://mlsanity.com/paper/2601.07219/) (2026)
- [U-CECE: A Universal Multi-Resolution Framework for Conceptual Counterfactual Explanations](https://mlsanity.com/paper/2604.08295/) (2026)
- [TBSG-Net: Temporal Bipartite Scene Graph Network for Fine-Grained Video Moment Retrieval](https://mlsanity.com/paper/2608.02056/) (2026)

## Abstract (arXiv)

As a scene graph compactly summarizes the high-level content of an image in a structured and symbolic manner, the similarity between scene graphs of two images reflects the relevance of their contents. Based on this idea, we propose a novel approach for image-to-image retrieval using scene graph similarity measured by graph neural networks. In our approach, graph neural networks are trained to predict the proxy image relevance measure, computed from human-annotated captions using a pre-trained sentence similarity model. We collect and publish the dataset for image relevance measured by human annotators to evaluate retrieval algorithms. The collected dataset shows that our method agrees well with the human perception of image similarity than other competitive baselines.