# DIPN: Deep Interaction Prediction Network with Application to Clutter Removal

arXiv: https://arxiv.org/abs/2011.04692 · HTML: https://mlsanity.com/paper/2011.04692/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Nov 2020.

## Built on by (most cited first, 7 of 7)

- [CaTGrasp: Learning Category-Level Task-Relevant Grasping in Clutter from Simulation](https://mlsanity.com/paper/2109.09163/) (2021)
- [Transporters with Visual Foresight for Solving Unseen Rearrangement Tasks](https://mlsanity.com/paper/2202.10765/) (2022)
- [Self-Supervised Interactive Object Segmentation Through a Singulation-and-Grasping Approach](https://mlsanity.com/paper/2207.09314/) (2022)
- [Efficient and High-quality Prehensile Rearrangement in Cluttered and Confined Spaces](https://mlsanity.com/paper/2110.02814/) (2021)
- [Toward Efficient Task Planning for Dual-Arm Tabletop Object Rearrangement](https://mlsanity.com/paper/2207.08078/) (2022)
- [ORLA*: Mobile Manipulator-Based Object Rearrangement with Lazy A Star](https://mlsanity.com/paper/2309.13707/) (2023)
- [Visual Prompt Guided Unified Pushing Policy](https://mlsanity.com/paper/2602.19193/) (2026)

## Abstract (arXiv)

We propose a Deep Interaction Prediction Network (DIPN) for learning to predict complex interactions that ensue as a robot end-effector pushes multiple objects, whose physical properties, including size, shape, mass, and friction coefficients may be unknown a priori. DIPN "imagines" the effect of a push action and generates an accurate synthetic image of the predicted outcome. DIPN is shown to be sample efficient when trained in simulation or with a real robotic system. The high accuracy of DIPN allows direct integration with a grasp network, yielding a robotic manipulation system capable of executing challenging clutter removal tasks while being trained in a fully self-supervised manner. The overall network demonstrates intelligent behavior in selecting proper actions between push and grasp for completing clutter removal tasks and significantly outperforms the previous state-of-the-art. Remarkably, DIPN achieves even better performance on the real robotic hardware system than in simulation. Videos, code, and experiments log are available at https://github.com/rutgers-arc-lab/dipn.