# Relation Regularized Scene Graph Generation

arXiv: https://arxiv.org/abs/2202.10826 · HTML: https://mlsanity.com/paper/2202.10826/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 47 references that are in the map.
- First posted on arXiv 22 Feb 2022.

## Builds on (most cited first, 30 of 34)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [Modeling Relational Data with Graph Convolutional Networks](https://mlsanity.com/paper/1703.06103/) (2017)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [Embedding Entities and Relations for Learning and Inference in Knowledge Bases](https://mlsanity.com/paper/1412.6575/) (2014)
- [Scene Graph Generation by Iterative Message Passing](https://mlsanity.com/paper/1701.02426/) (2017)
- [Neural Motifs: Scene Graph Parsing with Global Context](https://mlsanity.com/paper/1711.06640/) (2017)
- [Visual Relationship Detection with Language Priors](https://mlsanity.com/paper/1608.00187/) (2016)
- [Graph R-CNN for Scene Graph Generation](https://mlsanity.com/paper/1808.00191/) (2018)
- [Learning to Compose Dynamic Tree Structures for Visual Contexts](https://mlsanity.com/paper/1812.01880/) (2018)
- [Learning Human-Object Interactions by Graph Parsing Neural Networks](https://mlsanity.com/paper/1808.07962/) (2018)
- [Scene Graph Generation from Objects, Phrases and Region Captions](https://mlsanity.com/paper/1707.09700/) (2017)
- [Encoding Sentences with Graph Convolutional Networks for Semantic Role Labeling](https://mlsanity.com/paper/1703.04826/) (2017)
- [Visual Translation Embedding Network for Visual Relation Detection](https://mlsanity.com/paper/1702.08319/) (2017)
- [Detecting Visual Relationships with Deep Relational Networks](https://mlsanity.com/paper/1704.03114/) (2017)
- [Factorizable Net: An Efficient Subgraph-based Framework for Scene Graph Generation](https://mlsanity.com/paper/1806.11538/) (2018)
- [Pixels to Graphs by Associative Embedding](https://mlsanity.com/paper/1706.07365/) (2017)
- [Counterfactual Critic Multi-Agent Training for Scene Graph Generation](https://mlsanity.com/paper/1812.02347/) (2018)
- [Mapping Images to Scene Graphs with Permutation-Invariant Structured Prediction](https://mlsanity.com/paper/1802.05451/) (2018)
- [Zoom-Net: Mining Deep Feature Interactions for Visual Relationship Recognition](https://mlsanity.com/paper/1807.04979/) (2018)
- [Self-Supervised Video Hashing with Hierarchical Binary Auto-encoder](https://mlsanity.com/paper/1802.02305/) (2018)

## Built on by (most cited first, 3 of 3)

- [Learnable Graph Convolutional Network and Feature Fusion for Multi-view Learning](https://mlsanity.com/paper/2211.09155/) (2022)
- [KE-RCNN: Unifying Knowledge based Reasoning into Part-level Attribute Parsing](https://mlsanity.com/paper/2206.10146/) (2022)
- [SG-Shuffle: Multi-aspect Shuffle Transformer for Scene Graph Generation](https://mlsanity.com/paper/2211.04773/) (2022)

## Abstract (arXiv)

Scene graph generation (SGG) is built on top of detected objects to predict object pairwise visual relations for describing the image content abstraction. Existing works have revealed that if the links between objects are given as prior knowledge, the performance of SGG is significantly improved. Inspired by this observation, in this article, we propose a relation regularized network (R2-Net), which can predict whether there is a relationship between two objects and encode this relation into object feature refinement and better SGG. Specifically, we first construct an affinity matrix among detected objects to represent the probability of a relationship between two objects. Graph convolution networks (GCNs) over this relation affinity matrix are then used as object encoders, producing relation-regularized representations of objects. With these relation-regularized features, our R2-Net can effectively refine object labels and generate scene graphs. Extensive experiments are conducted on the visual genome dataset for three SGG tasks (i.e., predicate classification, scene graph classification, and scene graph detection), demonstrating the effectiveness of our proposed method. Ablation studies also verify the key roles of our proposed components in performance improvement.