# SafePicking: Learning Safe Object Extraction via Object-Level Mapping

arXiv: https://arxiv.org/abs/2202.05832 · HTML: https://mlsanity.com/paper/2202.05832/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 37 references that are in the map.
- First posted on arXiv 11 Feb 2022.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/1711.00199/) (2017)
- [QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation](https://mlsanity.com/paper/1806.10293/) (2018)
- [Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection](https://mlsanity.com/paper/1603.02199/) (2016)
- [Emergent Tool Use From Multi-Agent Autocurricula](https://mlsanity.com/paper/1909.07528/) (2019)
- [Deep Object Pose Estimation for Semantic Robotic Grasping of Household Objects](https://mlsanity.com/paper/1809.10790/) (2018)
- [Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hours](https://mlsanity.com/paper/1509.06825/) (2015)
- [kPAM: KeyPoint Affordances for Category-Level Robotic Manipulation](https://mlsanity.com/paper/1903.06684/) (2019)
- [TossingBot: Learning to Throw Arbitrary Objects with Residual Physics](https://mlsanity.com/paper/1903.11239/) (2019)
- [Learning Synergies between Pushing and Grasping with Self-supervised Deep Reinforcement Learning](https://mlsanity.com/paper/1803.09956/) (2018)
- [Robotic Pick-and-Place of Novel Objects in Clutter with Multi-Affordance Grasping and Cross-Domain Image Matching](https://mlsanity.com/paper/1710.01330/) (2017)
- [Coarse-to-Fine Q-attention: Efficient Learning for Visual Robotic Manipulation via Discretisation](https://mlsanity.com/paper/2106.12534/) (2021)
- [Intel RealSense Stereoscopic Depth Cameras](https://mlsanity.com/paper/1705.05548/) (2017)
- [Multi-view Self-supervised Deep Learning for 6D Pose Estimation in the Amazon Picking Challenge](https://mlsanity.com/paper/1609.09475/) (2016)
- [Form2Fit: Learning Shape Priors for Generalizable Assembly from Disassembly](https://mlsanity.com/paper/1910.13675/) (2019)
- [MoreFusion: Multi-object Reasoning for 6D Pose Estimation from Volumetric Fusion](https://mlsanity.com/paper/2004.04336/) (2020)
- [Multi-Task Domain Adaptation for Deep Learning of Instance Grasping from Simulation](https://mlsanity.com/paper/1710.06422/) (2017)
- [Grasp2Vec: Learning Object Representations from Self-Supervised Grasping](https://mlsanity.com/paper/1811.06964/) (2018)
- [kPAM-SC: Generalizable Manipulation Planning using KeyPoint Affordance and Shape Completion](https://mlsanity.com/paper/1909.06980/) (2019)
- [Visuomotor Mechanical Search: Learning to Retrieve Target Objects in Clutter](https://mlsanity.com/paper/2008.06073/) (2020)
- [GRIP: Generative Robust Inference and Perception for Semantic Robot Manipulation in Adversarial Environments](https://mlsanity.com/paper/1903.08352/) (2019)
- [GeoFusion: Geometric Consistency informed Scene Estimation in Dense Clutter](https://mlsanity.com/paper/2003.12610/) (2020)

## Built on by (most cited first, 3 of 3)

- [Sim-to-Real 6D Object Pose Estimation via Iterative Self-training for Robotic Bin Picking](https://mlsanity.com/paper/2204.07049/) (2022)
- [ReorientBot: Learning Object Reorientation for Specific-Posed Placement](https://mlsanity.com/paper/2202.11092/) (2022)
- [Coarse-to-fine Q-attention with Tree Expansion](https://mlsanity.com/paper/2204.12471/) (2022)

## Abstract (arXiv)

Robots need object-level scene understanding to manipulate objects while reasoning about contact, support, and occlusion among objects. Given a pile of objects, object recognition and reconstruction can identify the boundary of object instances, giving important cues as to how the objects form and support the pile. In this work, we present a system, SafePicking, that integrates object-level mapping and learning-based motion planning to generate a motion that safely extracts occluded target objects from a pile. Planning is done by learning a deep Q-network that receives observations of predicted poses and a depth-based heightmap to output a motion trajectory, trained to maximize a safety metric reward. Our results show that the observation fusion of poses and depth-sensing gives both better performance and robustness to the model. We evaluate our methods using the YCB objects in both simulation and the real world, achieving safe object extraction from piles.