# Neural Message Passing for Visual Relationship Detection

arXiv: https://arxiv.org/abs/2208.04165 · HTML: https://mlsanity.com/paper/2208.04165/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 37 references that are in the map.
- First posted on arXiv 8 Aug 2022.

## Builds on (most cited first, 28 of 28)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Relational inductive biases, deep learning, and graph networks](https://mlsanity.com/paper/1806.01261/) (2018)
- [Show and Tell: A Neural Image Caption Generator](https://mlsanity.com/paper/1411.4555/) (2014)
- [Visual Relationship Detection with Language Priors](https://mlsanity.com/paper/1608.00187/) (2016)
- [Graph R-CNN for Scene Graph Generation](https://mlsanity.com/paper/1808.00191/) (2018)
- [Learning Human-Object Interactions by Graph Parsing Neural Networks](https://mlsanity.com/paper/1808.07962/) (2018)
- [Graph HyperNetworks for Neural Architecture Search](https://mlsanity.com/paper/1810.05749/) (2018)
- [Visual Translation Embedding Network for Visual Relation Detection](https://mlsanity.com/paper/1702.08319/) (2017)
- [Detecting Visual Relationships with Deep Relational Networks](https://mlsanity.com/paper/1704.03114/) (2017)
- [Rethinking Knowledge Graph Propagation for Zero-Shot Learning](https://mlsanity.com/paper/1805.11724/) (2018)
- [Visual Relationship Detection with Internal and External Linguistic Knowledge Distillation](https://mlsanity.com/paper/1707.09423/) (2017)
- [LinkNet: Relational Embedding for Scene Graph](https://mlsanity.com/paper/1811.06410/) (2018)
- [Zoom-Net: Mining Deep Feature Interactions for Visual Relationship Recognition](https://mlsanity.com/paper/1807.04979/) (2018)
- [Situation Recognition with Graph Neural Networks](https://mlsanity.com/paper/1708.04320/) (2017)
- [Phrase Localization and Visual Relationship Detection with Comprehensive Image-Language Cues](https://mlsanity.com/paper/1611.06641/) (2016)
- [Deep Variation-structured Reinforcement Learning for Visual Relationship and Attribute Detection](https://mlsanity.com/paper/1703.03054/) (2017)
- [Conditional Neural Relational Inference for Interacting Systems](https://mlsanity.com/paper/2106.11083/) (2021)
- [Towards Context-aware Interaction Recognition](https://mlsanity.com/paper/1703.06246/) (2017)
- [Generalized Value Iteration Networks: Life Beyond Lattices](https://mlsanity.com/paper/1706.02416/) (2017)
- [Shuffle-Then-Assemble: Learning Object-Agnostic Visual Relationship Features](https://mlsanity.com/paper/1808.00171/) (2018)
- [Context-Dependent Diffusion Network for Visual Relationship Detection](https://mlsanity.com/paper/1809.06213/) (2018)

## Built on by (most cited first, 6 of 6)

- [The All-Seeing Project V2: Towards General Relation Comprehension of the Open World](https://mlsanity.com/paper/2402.19474/) (2024)
- [HiLo: Exploiting High Low Frequency Relations for Unbiased Panoptic Scene Graph Generation](https://mlsanity.com/paper/2303.15994/) (2023)
- [VLPrompt: Vision-Language Prompting for Panoptic Scene Graph Generation](https://mlsanity.com/paper/2311.16492/) (2023)
- [Grounding Scene Graphs on Natural Images via Visio-Lingual Message Passing](https://mlsanity.com/paper/2211.01969/) (2022)
- [A Novel Neural-symbolic System under Statistical Relational Learning](https://mlsanity.com/paper/2309.08931/) (2023)
- [Multi-Task Edge Prediction in Temporally-Dynamic Video Graphs](https://mlsanity.com/paper/2212.02875/) (2022)

## Abstract (arXiv)

Visual relationship detection aims to detect the interactions between objects in an image; however, this task suffers from combinatorial explosion due to the variety of objects and interactions. Since the interactions associated with the same object are dependent, we explore the dependency of interactions to reduce the search space. We explicitly model objects and interactions by an interaction graph and then propose a message-passing-style algorithm to propagate the contextual information. We thus call the proposed method neural message passing (NMP). We further integrate language priors and spatial cues to rule out unrealistic interactions and capture spatial interactions. Experimental results on two benchmark datasets demonstrate the superiority of our proposed method. Our code is available at https://github.com/PhyllisH/NMP.