# Scene Graph Generation with External Knowledge and Image Reconstruction

arXiv: https://arxiv.org/abs/1904.00560 · HTML: https://mlsanity.com/paper/1904.00560/

## Key facts

- Cited by 102 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Apr 2019.

## Built on by (most cited first, 30 of 102)

- [SceneGraphFusion: Incremental 3D Scene Graph Prediction from RGB-D Sequences](https://mlsanity.com/paper/2103.14898/) (2021)
- [Bipartite Graph Network with Adaptive Message Passing for Unbiased Scene Graph Generation](https://mlsanity.com/paper/2104.00308/) (2021)
- [A Comprehensive Survey of Scene Graphs: Generation and Application](https://mlsanity.com/paper/2104.01111/) (2021)
- [Panoptic Scene Graph Generation](https://mlsanity.com/paper/2207.11247/) (2022)
- [RelTR: Relation Transformer for Scene Graph Generation](https://mlsanity.com/paper/2201.11460/) (2022)
- [The All-Seeing Project V2: Towards General Relation Comprehension of the Open World](https://mlsanity.com/paper/2402.19474/) (2024)
- [Scene Graph Generation: A Comprehensive Survey](https://mlsanity.com/paper/2201.00443/) (2022)
- [Learning to Compose Visual Relations](https://mlsanity.com/paper/2111.09297/) (2021)
- [Recovering the Unbiased Scene Graphs from the Biased Ones](https://mlsanity.com/paper/2107.02112/) (2021)
- [Learning of Visual Relations: The Devil is in the Tails](https://mlsanity.com/paper/2108.09668/) (2021)
- [Fully Convolutional Scene Graph Generation](https://mlsanity.com/paper/2103.16083/) (2021)
- [Semantics for Robotic Mapping, Perception and Interaction: A Survey](https://mlsanity.com/paper/2101.00443/) (2021)
- [From General to Specific: Informative Scene Graph Generation via Balance Adjustment](https://mlsanity.com/paper/2108.13129/) (2021)
- [Deep Learning Approaches on Image Captioning: A Review](https://mlsanity.com/paper/2201.12944/) (2022)
- [Incremental 3D Semantic Scene Graph Prediction from RGB Sequences](https://mlsanity.com/paper/2305.02743/) (2023)
- [Latent Graph Representations for Critical View of Safety Assessment](https://mlsanity.com/paper/2212.04155/) (2022)
- [Structure-CLIP: Towards Scene Graph Knowledge to Enhance Multi-modal Structured Representations](https://mlsanity.com/paper/2305.06152/) (2023)
- [CoTDet: Affordance Knowledge Prompting for Task Driven Object Detection](https://mlsanity.com/paper/2309.01093/) (2023)
- [Probabilistic Modeling of Semantic Ambiguity for Scene Graph Generation](https://mlsanity.com/paper/2103.05271/) (2021)
- [Towards Data-and Knowledge-Driven Artificial Intelligence: A Survey on Neuro-Symbolic Computing](https://mlsanity.com/paper/2210.15889/) (2022)
- [In Defense of Scene Graphs for Image Captioning](https://mlsanity.com/paper/2102.04990/) (2021)
- [VAEL: Bridging Variational Autoencoders and Probabilistic Logic Programming](https://mlsanity.com/paper/2202.04178/) (2022)
- [Linguistic Structures as Weak Supervision for Visual Scene Graph Generation](https://mlsanity.com/paper/2105.13994/) (2021)
- [Consensus Graph Representation Learning for Better Grounded Image Captioning](https://mlsanity.com/paper/2112.00974/) (2021)
- [Coarse-to-Fine Reasoning for Visual Question Answering](https://mlsanity.com/paper/2110.02526/) (2021)
- [Causal Reasoning Meets Visual Representation Learning: A Prospective Study](https://mlsanity.com/paper/2204.12037/) (2022)
- [Devil's on the Edges: Selective Quad Attention for Scene Graph Generation](https://mlsanity.com/paper/2304.03495/) (2023)
- [Resistance Training using Prior Bias: toward Unbiased Scene Graph Generation](https://mlsanity.com/paper/2201.06794/) (2022)
- [Zero-Shot Scene Graph Relation Prediction through Commonsense Knowledge Integration](https://mlsanity.com/paper/2107.05080/) (2021)
- [Not All Relations are Equal: Mining Informative Labels for Scene Graph Generation](https://mlsanity.com/paper/2111.13517/) (2021)

## Abstract (arXiv)

Scene graph generation has received growing attention with the advancements in image understanding tasks such as object detection, attributes and relationship prediction,~etc. However, existing datasets are biased in terms of object and relationship labels, or often come with noisy and missing annotations, which makes the development of a reliable scene graph prediction model very challenging. In this paper, we propose a novel scene graph generation algorithm with external knowledge and image reconstruction loss to overcome these dataset issues. In particular, we extract commonsense knowledge from the external knowledge base to refine object and phrase features for improving generalizability in scene graph generation. To address the bias of noisy object annotations, we introduce an auxiliary image reconstruction path to regularize the scene graph generation network. Extensive experiments show that our framework can generate better scene graphs, achieving the state-of-the-art performance on two benchmark datasets: Visual Relationship Detection and Visual Genome datasets.