# Neural Motifs: Scene Graph Parsing with Global Context

arXiv: https://arxiv.org/abs/1711.06640 · HTML: https://mlsanity.com/paper/1711.06640/

## Key facts

- Cited by 414 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Nov 2017.

## Built on by (most cited first, 30 of 414)

- [MERLOT Reserve: Neural Script Knowledge through Vision and Language and Sound](https://mlsanity.com/paper/2201.02639/) (2022)
- [CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models](https://mlsanity.com/paper/2109.11797/) (2021)
- [Bipartite Graph Network with Adaptive Message Passing for Unbiased Scene Graph Generation](https://mlsanity.com/paper/2104.00308/) (2021)
- [A Comprehensive Survey of Scene Graphs: Generation and Application](https://mlsanity.com/paper/2104.01111/) (2021)
- [Panoptic Scene Graph Generation](https://mlsanity.com/paper/2207.11247/) (2022)
- [Energy-Based Learning for Scene Graph Generation](https://mlsanity.com/paper/2103.02221/) (2021)
- [SkySenseGPT: A Fine-Grained Instruction Tuning Dataset and Model for Remote Sensing Vision-Language Understanding](https://mlsanity.com/paper/2406.10100/) (2024)
- [SGTR: End-to-end Scene Graph Generation with Transformer](https://mlsanity.com/paper/2112.12970/) (2021)
- [Spatial-Temporal Transformer for Dynamic Scene Graph Generation](https://mlsanity.com/paper/2107.12309/) (2021)
- [RelTR: Relation Transformer for Scene Graph Generation](https://mlsanity.com/paper/2201.11460/) (2022)
- [Semantic Communications: Principles and Challenges](https://mlsanity.com/paper/2201.01389/) (2021)
- [The All-Seeing Project V2: Towards General Relation Comprehension of the Open World](https://mlsanity.com/paper/2402.19474/) (2024)
- [TransRefer3D: Entity-and-Relation Aware Transformer for Fine-Grained 3D Visual Grounding](https://mlsanity.com/paper/2108.02388/) (2021)
- [Graph-to-3D: End-to-End Generation and Manipulation of 3D Scenes Using Scene Graphs](https://mlsanity.com/paper/2108.08841/) (2021)
- [Multi-Modal Knowledge Graph Construction and Application: A Survey](https://mlsanity.com/paper/2202.05786/) (2022)
- [Scene Graph Generation: A Comprehensive Survey](https://mlsanity.com/paper/2201.00443/) (2022)
- [Recovering the Unbiased Scene Graphs from the Biased Ones](https://mlsanity.com/paper/2107.02112/) (2021)
- [Stacked Hybrid-Attention and Group Collaborative Learning for Unbiased Scene Graph Generation](https://mlsanity.com/paper/2203.09811/) (2022)
- [Relationformer: A Unified Framework for Image-to-Graph Generation](https://mlsanity.com/paper/2203.10202/) (2022)
- [Learning of Visual Relations: The Devil is in the Tails](https://mlsanity.com/paper/2108.09668/) (2021)
- [Learning to Generate Scene Graph from Natural Language Supervision](https://mlsanity.com/paper/2109.02227/) (2021)
- [The Devil is in the Labels: Noisy Label Correction for Robust Scene Graph Generation](https://mlsanity.com/paper/2206.03014/) (2022)
- [Fully Convolutional Scene Graph Generation](https://mlsanity.com/paper/2103.16083/) (2021)
- [Fine-Grained Scene Graph Generation with Data Transfer](https://mlsanity.com/paper/2203.11654/) (2022)
- [Semantics for Robotic Mapping, Perception and Interaction: A Survey](https://mlsanity.com/paper/2101.00443/) (2021)
- [A Survey on Graph Neural Networks and Graph Transformers in Computer Vision: A Task-Oriented Perspective](https://mlsanity.com/paper/2209.13232/) (2022)
- [Towards Open-vocabulary Scene Graph Generation with Prompt-based Finetuning](https://mlsanity.com/paper/2208.08165/) (2022)
- [Target Adaptive Context Aggregation for Video Scene Graph Generation](https://mlsanity.com/paper/2108.08121/) (2021)
- [From General to Specific: Informative Scene Graph Generation via Balance Adjustment](https://mlsanity.com/paper/2108.13129/) (2021)
- [FACTUAL: A Benchmark for Faithful and Consistent Textual Scene Graph Parsing](https://mlsanity.com/paper/2305.17497/) (2023)

## Abstract (arXiv)

We investigate the problem of producing structured graph representations of visual scenes. Our work analyzes the role of motifs: regularly appearing substructures in scene graphs. We present new quantitative insights on such repeated structures in the Visual Genome dataset. Our analysis shows that object labels are highly predictive of relation labels but not vice-versa. We also find that there are recurring patterns even in larger subgraphs: more than 50% of graphs contain motifs involving at least two relations. Our analysis motivates a new baseline: given object detections, predict the most frequent relation between object pairs with the given labels, as seen in the training set. This baseline improves on the previous state-of-the-art by an average of 3.6% relative improvement across evaluation settings. We then introduce Stacked Motif Networks, a new architecture designed to capture higher order motifs in scene graphs that further improves over our strong baseline by an average 7.1% relative gain. Our code is available at github.com/rowanz/neural-motifs.