# Visual Manipulation Relationship Network

arXiv: https://arxiv.org/abs/1802.08857 · HTML: https://mlsanity.com/paper/1802.08857/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Feb 2018.

## Built on by (most cited first, 10 of 10)

- [INVIGORATE: Interactive Visual Grounding and Grasping in Clutter](https://mlsanity.com/paper/2108.11092/) (2021)
- [REGRAD: A Large-Scale Relational Grasp Dataset for Safe and Object-Specific Robotic Grasping in Clutter](https://mlsanity.com/paper/2104.14118/) (2021)
- [Free-form language-based robotic reasoning and grasping](https://mlsanity.com/paper/2503.13082/) (2025)
- [I Know What You Draw: Learning Grasp Detection Conditioned on a Few Freehand Sketches](https://mlsanity.com/paper/2205.04026/) (2022)
- [MetaGraspNet_v0: A Large-Scale Benchmark Dataset for Vision-driven Robotic Grasping via Physics-based Metaverse Synthesis](https://mlsanity.com/paper/2112.14663/) (2021)
- [SInViG: A Self-Evolving Interactive Visual Agent for Human-Robot Interaction](https://mlsanity.com/paper/2402.11792/) (2024)
- [A Modern Take on Visual Relationship Reasoning for Grasp Planning](https://mlsanity.com/paper/2409.02035/) (2024)
- [MMRDN: Consistent Representation for Multi-View Manipulation Relationship Detection in Object-Stacked Scenes](https://mlsanity.com/paper/2304.12592/) (2023)
- [Obstruction reasoning for robotic grasping](https://mlsanity.com/paper/2511.23186/) (2025)
- [GraspFactory: A Large Object-Centric Grasping Dataset](https://mlsanity.com/paper/2509.20550/) (2025)

## Abstract (arXiv)

Robotic grasping detection is one of the most important fields in robotics, in which great progress has been made recent years with the help of convolutional neural network (CNN). However, including multiple objects in one scene can invalidate the existing CNN-based grasping detection algorithms, because manipulation relationships among objects are not considered, which are required to guide the robot to grasp things in the right order. This paper presents a new CNN architecture called Visual Manipulation Relationship Network (VMRN) to help robot detect targets and predict the manipulation relationships in real time. To implement end-to-end training and meet real-time requirements in robot tasks, we propose the Object Pairing Pooling Layer (OP2L) to help to predict all manipulation relationships in one forward process. Moreover, in order to train VMRN, we collect a dataset named Visual Manipulation Relationship Dataset (VMRD) consisting of 5185 images with more than 17000 object instances and the manipulation relationships between all possible pairs of objects in every image, which is labeled by the manipulation relationship tree. The experimental results show that the new network architecture can detect objects and predict manipulation relationships simultaneously and meet the real-time requirements in robot tasks.