# Bridging Knowledge Graphs to Generate Scene Graphs

arXiv: https://arxiv.org/abs/2001.02314 · HTML: https://mlsanity.com/paper/2001.02314/

## Key facts

- Cited by 86 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Jan 2020.

## Built on by (most cited first, 30 of 86)

- [Bipartite Graph Network with Adaptive Message Passing for Unbiased Scene Graph Generation](https://mlsanity.com/paper/2104.00308/) (2021)
- [A Comprehensive Survey of Scene Graphs: Generation and Application](https://mlsanity.com/paper/2104.01111/) (2021)
- [Panoptic Scene Graph Generation](https://mlsanity.com/paper/2207.11247/) (2022)
- [Energy-Based Learning for Scene Graph Generation](https://mlsanity.com/paper/2103.02221/) (2021)
- [SGTR: End-to-end Scene Graph Generation with Transformer](https://mlsanity.com/paper/2112.12970/) (2021)
- [Learning to Predict Visual Attributes in the Wild](https://mlsanity.com/paper/2106.09707/) (2021)
- [RelTR: Relation Transformer for Scene Graph Generation](https://mlsanity.com/paper/2201.11460/) (2022)
- [Scene Graph Generation: A Comprehensive Survey](https://mlsanity.com/paper/2201.00443/) (2022)
- [Recovering the Unbiased Scene Graphs from the Biased Ones](https://mlsanity.com/paper/2107.02112/) (2021)
- [Stacked Hybrid-Attention and Group Collaborative Learning for Unbiased Scene Graph Generation](https://mlsanity.com/paper/2203.09811/) (2022)
- [Relationformer: A Unified Framework for Image-to-Graph Generation](https://mlsanity.com/paper/2203.10202/) (2022)
- [Learning of Visual Relations: The Devil is in the Tails](https://mlsanity.com/paper/2108.09668/) (2021)
- [Fully Convolutional Scene Graph Generation](https://mlsanity.com/paper/2103.16083/) (2021)
- [Fine-Grained Scene Graph Generation with Data Transfer](https://mlsanity.com/paper/2203.11654/) (2022)
- [Towards Open-vocabulary Scene Graph Generation with Prompt-based Finetuning](https://mlsanity.com/paper/2208.08165/) (2022)
- [From General to Specific: Informative Scene Graph Generation via Balance Adjustment](https://mlsanity.com/paper/2108.13129/) (2021)
- [VL-SAT: Visual-Linguistic Semantics Assisted Training for 3D Semantic Scene Graph Prediction in Point Cloud](https://mlsanity.com/paper/2303.14408/) (2023)
- [From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Models](https://mlsanity.com/paper/2404.00906/) (2024)
- [CoTDet: Affordance Knowledge Prompting for Task Driven Object Detection](https://mlsanity.com/paper/2309.01093/) (2023)
- [TIDEE: Tidying Up Novel Rooms using Visuo-Semantic Commonsense Priors](https://mlsanity.com/paper/2207.10761/) (2022)
- [Segmentation-grounded Scene Graph Generation](https://mlsanity.com/paper/2104.14207/) (2021)
- [Indoor and Outdoor 3D Scene Graph Generation via Language-Enabled Spatial Ontologies](https://mlsanity.com/paper/2312.11713/) (2023)
- [Iterative Scene Graph Generation](https://mlsanity.com/paper/2207.13440/) (2022)
- [Devil's on the Edges: Selective Quad Attention for Scene Graph Generation](https://mlsanity.com/paper/2304.03495/) (2023)
- [Not All Relations are Equal: Mining Informative Labels for Scene Graph Generation](https://mlsanity.com/paper/2111.13517/) (2021)
- [BGT-Net: Bidirectional GRU Transformer Network for Scene Graph Generation](https://mlsanity.com/paper/2109.05346/) (2021)
- [Vision Relation Transformer for Unbiased Scene Graph Generation](https://mlsanity.com/paper/2308.09472/) (2023)
- [From SLAM to Situational Awareness: Challenges and Survey](https://mlsanity.com/paper/2110.00273/) (2021)
- [Adaptive Fine-Grained Predicates Learning for Scene Graph Generation](https://mlsanity.com/paper/2207.04602/) (2022)
- [HiKER-SGG: Hierarchical Knowledge Enhanced Robust Scene Graph Generation](https://mlsanity.com/paper/2403.12033/) (2024)

## Abstract (arXiv)

Scene graphs are powerful representations that parse images into their abstract semantic elements, i.e., objects and their interactions, which facilitates visual comprehension and explainable reasoning. On the other hand, commonsense knowledge graphs are rich repositories that encode how the world is structured, and how general concepts interact. In this paper, we present a unified formulation of these two constructs, where a scene graph is seen as an image-conditioned instantiation of a commonsense knowledge graph. Based on this new perspective, we re-formulate scene graph generation as the inference of a bridge between the scene and commonsense graphs, where each entity or predicate instance in the scene graph has to be linked to its corresponding entity or predicate class in the commonsense graph. To this end, we propose a novel graph-based neural network that iteratively propagates information between the two graphs, as well as within each of them, while gradually refining their bridge in each iteration. Our Graph Bridging Network, GB-Net, successively infers edges and nodes, allowing to simultaneously exploit and refine the rich, heterogeneous structure of the interconnected scene and commonsense graphs. Through extensive experimentation, we showcase the superior accuracy of GB-Net compared to the most recent methods, resulting in a new state of the art. We publicly release the source code of our method.