# Predicting and Attending to Damaging Collisions for Placing Everyday Objects in Photo-Realistic Simulations

arXiv: https://arxiv.org/abs/2102.06507 · HTML: https://mlsanity.com/paper/2102.06507/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 35 references that are in the map.
- First posted on arXiv 12 Feb 2021.

## Builds on (most cited first, 16 of 16)

- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [PHYRE: A New Benchmark for Physical Reasoning](https://mlsanity.com/paper/1908.05656/) (2019)
- [Learning Rich Features from RGB-D Images for Object Detection and Segmentation](https://mlsanity.com/paper/1407.5736/) (2014)
- [Robotic Pick-and-Place of Novel Objects in Clutter with Multi-Affordance Grasping and Cross-Domain Image Matching](https://mlsanity.com/paper/1710.01330/) (2017)
- [Attention Branch Network: Learning of Attention Mechanism for Visual Explanation](https://mlsanity.com/paper/1812.10025/) (2018)
- [Attention-Based Multimodal Fusion for Video Description](https://mlsanity.com/paper/1701.03126/) (2017)
- ["What happens if..." Learning to Predict the Effect of Forces in Images](https://mlsanity.com/paper/1603.05600/) (2016)
- [Understanding Natural Language Instructions for Fetching Daily Objects Using GAN-Based Multimodal Target-Source Classification](https://mlsanity.com/paper/1906.06830/) (2019)
- [Pick and Place Without Geometric Object Models](https://mlsanity.com/paper/1707.05615/) (2017)
- [A Multimodal Classifier Generative Adversarial Network for Carry and Place Tasks from Ambiguous Language Instructions](https://mlsanity.com/paper/1806.03847/) (2018)
- [Plane-Based Optimization of Geometry and Texture for RGB-D Reconstruction of Indoor Scenes](https://mlsanity.com/paper/1905.09829/) (2019)
- [A Multimodal Target-Source Classifier with Attention Branches to Understand Ambiguous Instructions for Fetching Daily Objects](https://mlsanity.com/paper/1912.10675/) (2019)
- [Multimodal Attention Branch Network for Perspective-Free Sentence Generation](https://mlsanity.com/paper/1909.05664/) (2019)

## Built on by (most cited first, 2 of 2)

- [Relational Future Captioning Model for Explaining Likely Collisions in Daily Tasks](https://mlsanity.com/paper/2207.09083/) (2022)
- [Attention Lattice Adapter: Visual Explanation Generation for Visual Foundation Model](https://mlsanity.com/paper/2509.14664/) (2025)

## Abstract (arXiv)

Placing objects is a fundamental task for domestic service robots (DSRs). Thus, inferring the collision-risk before a placing motion is crucial for achieving the requested task. This problem is particularly challenging because it is necessary to predict what happens if an object is placed in a cluttered designated area. We show that a rule-based approach that uses plane detection, to detect free areas, performs poorly. To address this, we develop PonNet, which has multimodal attention branches and a self-attention mechanism to predict damaging collisions, based on RGBD images. Our method can visualize the risk of damaging collisions, which is convenient because it enables the user to understand the risk. For this purpose, we build and publish an original dataset that contains 12,000 photo-realistic images of specific placing areas, with daily life objects, in home environments. The experimental results show that our approach improves accuracy compared with the baseline methods.