# Image Scene Graph Generation (SGG) Benchmark

arXiv: https://arxiv.org/abs/2107.12604 · HTML: https://mlsanity.com/paper/2107.12604/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 23 references that are in the map.
- First posted on arXiv 27 Jul 2021.

## Builds on (most cited first, 18 of 18)

- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [Scene Graph Generation by Iterative Message Passing](https://mlsanity.com/paper/1701.02426/) (2017)
- [Neural Motifs: Scene Graph Parsing with Global Context](https://mlsanity.com/paper/1711.06640/) (2017)
- [Image Generation from Scene Graphs](https://mlsanity.com/paper/1804.01622/) (2018)
- [Graph R-CNN for Scene Graph Generation](https://mlsanity.com/paper/1808.00191/) (2018)
- [3D Scene Graph: A Structure for Unified Semantics, 3D Space, and Camera](https://mlsanity.com/paper/1910.02527/) (2019)
- [Learning to Compose Dynamic Tree Structures for Visual Contexts](https://mlsanity.com/paper/1812.01880/) (2018)
- [Auto-Encoding Scene Graphs for Image Captioning](https://mlsanity.com/paper/1812.02378/) (2018)
- [Action Genome: Actions as Composition of Spatio-temporal Scene Graphs](https://mlsanity.com/paper/1912.06992/) (2019)
- [Scene Graph Generation from Objects, Phrases and Region Captions](https://mlsanity.com/paper/1707.09700/) (2017)
- [VinVL: Revisiting Visual Representations in Vision-Language Models](https://mlsanity.com/paper/2101.00529/) (2021)
- [Graphical Contrastive Losses for Scene Graph Parsing](https://mlsanity.com/paper/1903.02728/) (2019)
- [Unpaired Image Captioning via Scene Graph Alignments](https://mlsanity.com/paper/1903.10658/) (2019)
- [Cross-modal Scene Graph Matching for Relationship-aware Image-Text Retrieval](https://mlsanity.com/paper/1910.05134/) (2019)
- [BLOCK: Bilinear Superdiagonal Fusion for Visual Question Answering and Visual Relationship Detection](https://mlsanity.com/paper/1902.00038/) (2019)
- [Object-Centric Image Generation from Layouts](https://mlsanity.com/paper/2003.07449/) (2020)
- [Generating Natural Language Explanations for Visual Question Answering using Scene Graphs and Visual Attention](https://mlsanity.com/paper/1902.05715/) (2019)

## Built on by (most cited first, 15 of 15)

- [Grounded Language-Image Pre-training](https://mlsanity.com/paper/2112.03857/) (2021)
- [Towards Open-vocabulary Scene Graph Generation with Prompt-based Finetuning](https://mlsanity.com/paper/2208.08165/) (2022)
- [See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoning](https://mlsanity.com/paper/2301.05226/) (2023)
- [Measuring Representational Harms in Image Captioning](https://mlsanity.com/paper/2206.07173/) (2022)
- [A Thousand Words Are Worth More Than a Picture: Natural Language-Centric Outside-Knowledge Visual Question Answering](https://mlsanity.com/paper/2201.05299/) (2022)
- [Multitask Multimodal Prompted Training for Interactive Embodied Task Completion](https://mlsanity.com/paper/2311.04067/) (2023)
- [GeReA: Question-Aware Prompt Captions for Knowledge-based Visual Question Answering](https://mlsanity.com/paper/2402.02503/) (2024)
- [Quality-agnostic Image Captioning to Safely Assist People with Vision Impairment](https://mlsanity.com/paper/2304.14623/) (2023)
- [Towards Lifelong Scene Graph Generation with Knowledge-ware In-context Prompt Learning](https://mlsanity.com/paper/2401.14626/) (2024)
- [CLID: Controlled-Length Image Descriptions with Limited Data](https://mlsanity.com/paper/2211.14835/) (2022)
- [Towards Addressing the Misalignment of Object Proposal Evaluation for Vision-Language Tasks via Semantic Grounding](https://mlsanity.com/paper/2309.00215/) (2023)
- [CLOP: Video-and-Language Pre-Training with Knowledge Regularizations](https://mlsanity.com/paper/2211.03314/) (2022)
- [On Utilizing Relationships for Transferable Few-Shot Fine-Grained Object Detection](https://mlsanity.com/paper/2212.00770/) (2022)
- [The Solution for the CVPR 2023 1st foundation model challenge-Track2](https://mlsanity.com/paper/2403.17702/) (2024)
- [Enhanced Data Transfer Cooperating with Artificial Triplets for Scene Graph Generation](https://mlsanity.com/paper/2406.19316/) (2024)

## Abstract (arXiv)

There is a surge of interest in image scene graph generation (object, attribute and relationship detection) due to the need of building fine-grained image understanding models that go beyond object detection. Due to the lack of a good benchmark, the reported results of different scene graph generation models are not directly comparable, impeding the research progress. We have developed a much-needed scene graph generation benchmark based on the maskrcnn-benchmark and several popular models. This paper presents main features of our benchmark and a comprehensive ablation study of scene graph generation models using the Visual Genome and OpenImages Visual relationship detection datasets. Our codebase is made publicly available at https://github.com/microsoft/scene_graph_benchmark.