# Unbiased Heterogeneous Scene Graph Generation with Relation-aware Message Passing Neural Network

arXiv: https://arxiv.org/abs/2212.00443 · HTML: https://mlsanity.com/paper/2212.00443/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 52 references that are in the map.
- First posted on arXiv 1 Dec 2022.

## Builds on (most cited first, 30 of 35)

- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Modeling Relational Data with Graph Convolutional Networks](https://mlsanity.com/paper/1703.06103/) (2017)
- [Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning](https://mlsanity.com/paper/1801.07606/) (2018)
- [Heterogeneous Graph Attention Network](https://mlsanity.com/paper/1903.07293/) (2019)
- [Heterogeneous Graph Transformer](https://mlsanity.com/paper/2003.01332/) (2020)
- [The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale](https://mlsanity.com/paper/1811.00982/) (2018)
- [Scene Graph Generation by Iterative Message Passing](https://mlsanity.com/paper/1701.02426/) (2017)
- [Neural Motifs: Scene Graph Parsing with Global Context](https://mlsanity.com/paper/1711.06640/) (2017)
- [Unbiased Scene Graph Generation from Biased Training](https://mlsanity.com/paper/2002.11949/) (2020)
- [Graph R-CNN for Scene Graph Generation](https://mlsanity.com/paper/1808.00191/) (2018)
- [Learning to Compose Dynamic Tree Structures for Visual Contexts](https://mlsanity.com/paper/1812.01880/) (2018)
- [Auto-Encoding Scene Graphs for Image Captioning](https://mlsanity.com/paper/1812.02378/) (2018)
- [GPS-Net: Graph Property Sensing Network for Scene Graph Generation](https://mlsanity.com/paper/2003.12962/) (2020)
- [Knowledge-Embedded Routing Network for Scene Graph Generation](https://mlsanity.com/paper/1903.03326/) (2019)
- [Scene Graph Generation from Objects, Phrases and Region Captions](https://mlsanity.com/paper/1707.09700/) (2017)
- [Bipartite Graph Network with Adaptive Message Passing for Unbiased Scene Graph Generation](https://mlsanity.com/paper/2104.00308/) (2021)
- [Graphical Contrastive Losses for Scene Graph Parsing](https://mlsanity.com/paper/1903.02728/) (2019)
- [Unsupervised Attributed Multiplex Network Embedding](https://mlsanity.com/paper/1911.06750/) (2019)
- [HDMI: High-order Deep Multiplex Infomax](https://mlsanity.com/paper/2102.07810/) (2021)
- [PCPL: Predicate-Correlation Perception Learning for Unbiased Scene Graph Generation](https://mlsanity.com/paper/2009.00893/) (2020)
- [Attentive Relational Networks for Mapping Images to Scene Graphs](https://mlsanity.com/paper/1811.10696/) (2018)
- [Recovering the Unbiased Scene Graphs from the Biased Ones](https://mlsanity.com/paper/2107.02112/) (2021)
- [Stacked Hybrid-Attention and Group Collaborative Learning for Unbiased Scene Graph Generation](https://mlsanity.com/paper/2203.09811/) (2022)
- [Learning of Visual Relations: The Devil is in the Tails](https://mlsanity.com/paper/2108.09668/) (2021)
- [The Devil is in the Labels: Noisy Label Correction for Robust Scene Graph Generation](https://mlsanity.com/paper/2206.03014/) (2022)
- [Structured Query-Based Image Retrieval Using Scene Graphs](https://mlsanity.com/paper/2005.06653/) (2020)
- [Graph Density-Aware Losses for Novel Compositions in Scene Graph Generation](https://mlsanity.com/paper/2005.08230/) (2020)
- [Generating Natural Language Explanations for Visual Question Answering using Scene Graphs and Visual Attention](https://mlsanity.com/paper/1902.05715/) (2019)

## Built on by (most cited first, 17 of 17)

- [STAR: A First-Ever Dataset and A Large-Scale Benchmark for Scene Graph Generation in Large-Size Satellite Imagery](https://mlsanity.com/paper/2406.09410/) (2024)
- [LLM4SGG: Large Language Models for Weakly Supervised Scene Graph Generation](https://mlsanity.com/paper/2310.10404/) (2023)
- [A Causal Adjustment Module for Debiasing Scene Graph Generation](https://mlsanity.com/paper/2503.17862/) (2025)
- [Adaptive Self-training Framework for Fine-grained Scene Graph Generation](https://mlsanity.com/paper/2401.09786/) (2024)
- [DSGG: Dense Relation Transformer for an End-to-end Scene Graph Generation](https://mlsanity.com/paper/2403.14886/) (2024)
- [RA-SGG: Retrieval-Augmented Scene Graph Generation Framework via Multi-Prototype Learning](https://mlsanity.com/paper/2412.12788/) (2024)
- [Fine-Grained is Too Coarse: A Novel Data-Centric Approach for Efficient Scene Graph Generation](https://mlsanity.com/paper/2305.18668/) (2023)
- [Scene Graph Generation Strategy with Co-occurrence Knowledge and Learnable Term Frequency](https://mlsanity.com/paper/2405.12648/) (2024)
- [Semantic Diversity-aware Prototype-based Learning for Unbiased Scene Graph Generation](https://mlsanity.com/paper/2407.15396/) (2024)
- [Weakly Supervised Video Scene Graph Generation via Natural Language Supervision](https://mlsanity.com/paper/2502.15370/) (2025)
- [Semantic Scene Graph Generation Based on an Edge Dual Scene Graph and Message Passing Neural Network](https://mlsanity.com/paper/2311.01192/) (2023)
- [MSG-Chart: Multimodal Scene Graph for ChartQA](https://mlsanity.com/paper/2408.04852/) (2024)
- [A Reverse Causal Framework to Mitigate Spurious Correlations for Debiasing Scene Graph Generation](https://mlsanity.com/paper/2505.23451/) (2025)
- [Ensemble Predicate Decoding for Unbiased Scene Graph Generation](https://mlsanity.com/paper/2408.14187/) (2024)
- [Unbiased Scene Graph Generation by Type-Aware Message Passing on Heterogeneous and Dual Graphs](https://mlsanity.com/paper/2411.13287/) (2024)
- [Leveraging Static Relationships for Intra-Type and Inter-Type Message Passing in Video Question Answering](https://mlsanity.com/paper/2504.02417/) (2025)
- [Learning Context-Conditioned Predicate Semantics via Prototype Feedback](https://mlsanity.com/paper/2605.29610/) (2026)

## Abstract (arXiv)

Recent scene graph generation (SGG) frameworks have focused on learning complex relationships among multiple objects in an image. Thanks to the nature of the message passing neural network (MPNN) that models high-order interactions between objects and their neighboring objects, they are dominant representation learning modules for SGG. However, existing MPNN-based frameworks assume the scene graph as a homogeneous graph, which restricts the context-awareness of visual relations between objects. That is, they overlook the fact that the relations tend to be highly dependent on the objects with which the relations are associated. In this paper, we propose an unbiased heterogeneous scene graph generation (HetSGG) framework that captures relation-aware context using message passing neural networks. We devise a novel message passing layer, called relation-aware message passing neural network (RMP), that aggregates the contextual information of an image considering the predicate type between objects. Our extensive evaluations demonstrate that HetSGG outperforms state-of-the-art methods, especially outperforming on tail predicate classes.