# ViP-CNN: Visual Phrase Guided Convolutional Neural Network

arXiv: https://arxiv.org/abs/1702.07191 · HTML: https://mlsanity.com/paper/1702.07191/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Feb 2017.

## Built on by (most cited first, 8 of 8)

- [RelTR: Relation Transformer for Scene Graph Generation](https://mlsanity.com/paper/2201.11460/) (2022)
- [Structured Sparse R-CNN for Direct Scene Graph Generation](https://mlsanity.com/paper/2106.10815/) (2021)
- [ACP++: Action Co-occurrence Priors for Human-Object Interaction Detection](https://mlsanity.com/paper/2109.04047/) (2021)
- [2.5D Visual Relationship Detection](https://mlsanity.com/paper/2104.12727/) (2021)
- [Skew Class-balanced Re-weighting for Unbiased Scene Graph Generation](https://mlsanity.com/paper/2301.00351/) (2023)
- [STUPD: A Synthetic Dataset for Spatial and Temporal Relation Reasoning](https://mlsanity.com/paper/2309.06680/) (2023)
- [ViRel: Unsupervised Visual Relations Discovery with Graph-level Analogy](https://mlsanity.com/paper/2207.00590/) (2022)
- [SGDraw: Scene Graph Drawing Interface Using Object-Oriented Representation](https://mlsanity.com/paper/2211.16697/) (2022)

## Abstract (arXiv)

As the intermediate level task connecting image captioning and object detection, visual relationship detection started to catch researchers' attention because of its descriptive power and clear structure. It detects the objects and captures their pair-wise interactions with a subject-predicate-object triplet, e.g. person-ride-horse. In this paper, each visual relationship is considered as a phrase with three components. We formulate the visual relationship detection as three inter-connected recognition problems and propose a Visual Phrase guided Convolutional Neural Network (ViP-CNN) to address them simultaneously. In ViP-CNN, we present a Phrase-guided Message Passing Structure (PMPS) to establish the connection among relationship components and help the model consider the three problems jointly. Corresponding non-maximum suppression method and model training strategy are also proposed. Experimental results show that our ViP-CNN outperforms the state-of-art method both in speed and accuracy. We further pretrain ViP-CNN on our cleansed Visual Genome Relationship dataset, which is found to perform better than the pretraining on the ImageNet for this task.