# Transferring Experience from Simulation to the Real World for Precise Pick-And-Place Tasks in Highly Cluttered Scenes

arXiv: https://arxiv.org/abs/2101.04781 · HTML: https://mlsanity.com/paper/2101.04781/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 44 references that are in the map.
- First posted on arXiv 12 Jan 2021.

## Builds on (most cited first, 27 of 27)

- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [YOLO9000: Better, Faster, Stronger](https://mlsanity.com/paper/1612.08242/) (2016)
- [QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation](https://mlsanity.com/paper/1806.10293/) (2018)
- [Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection](https://mlsanity.com/paper/1603.02199/) (2016)
- [Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics](https://mlsanity.com/paper/1703.09312/) (2017)
- [Deep Object Pose Estimation for Semantic Robotic Grasping of Household Objects](https://mlsanity.com/paper/1809.10790/) (2018)
- [Real-Time Seamless Single Shot 6D Object Pose Prediction](https://mlsanity.com/paper/1711.08848/) (2017)
- [Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hours](https://mlsanity.com/paper/1509.06825/) (2015)
- [Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via Randomized-to-Canonical Adaptation Networks](https://mlsanity.com/paper/1812.07252/) (2018)
- [Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic Grasping](https://mlsanity.com/paper/1709.07857/) (2017)
- [Closing the Loop for Robotic Grasping: A Real-time, Generative Grasp Synthesis Approach](https://mlsanity.com/paper/1804.05172/) (2018)
- [Deep Learning for Detecting Robotic Grasps](https://mlsanity.com/paper/1301.3592/) (2013)
- [Data-Driven Grasp Synthesis - A Survey](https://mlsanity.com/paper/1309.2660/) (2013)
- [Jacquard: A Large Scale Dataset for Robotic Grasp Detection](https://mlsanity.com/paper/1803.11469/) (2018)
- [Real-Time Grasp Detection Using Convolutional Neural Networks](https://mlsanity.com/paper/1412.3128/) (2014)
- [Faster k-Medoids Clustering: Improving the PAM, CLARA, and CLARANS Algorithms](https://mlsanity.com/paper/1810.05691/) (2018)
- [Large-scale 6D Object Pose Estimation Dataset for Industrial Bin-Picking](https://mlsanity.com/paper/1912.12125/) (2019)
- [Single Shot 6D Object Pose Estimation](https://mlsanity.com/paper/2004.12729/) (2020)
- [Defining the Pose of any 3D Rigid Object and an Associated Distance](https://mlsanity.com/paper/1612.04631/) (2016)
- [Dex-Net 3.0: Computing Robust Robot Vacuum Suction Grasp Targets in Point Clouds using a New Analytic Model and Deep Learning](https://mlsanity.com/paper/1709.06670/) (2017)
- [Multi-View Picking: Next-best-view Reaching for Improved Grasping in Clutter](https://mlsanity.com/paper/1809.08564/) (2018)
- [Symmetry Aware Evaluation of 3D Object Detection and Pose Estimation in Scenes of Many Parts in Bulk](https://mlsanity.com/paper/1806.08129/) (2018)
- [Real-Time 6D Object Pose Estimation on CPU](https://mlsanity.com/paper/1811.08588/) (2018)
- [Improving Data Efficiency of Self-supervised Learning for Robotic Grasping](https://mlsanity.com/paper/1903.00228/) (2019)

## Built on by (most cited first, 6 of 6)

- [MetaGraspNet: A Large-Scale Benchmark Dataset for Scene-Aware Ambidextrous Bin Picking via Physics-based Metaverse Synthesis](https://mlsanity.com/paper/2208.03963/) (2022)
- [Cluttered Food Grasping with Adaptive Fingers and Synthetic-Data Trained Object Detection](https://mlsanity.com/paper/2203.05187/) (2022)
- [Sim-to-Real Grasp Detection with Global-to-Local RGB-D Adaptation](https://mlsanity.com/paper/2403.11511/) (2024)
- [Investigations on Output Parameterizations of Neural Networks for Single Shot 6D Object Pose Estimation](https://mlsanity.com/paper/2104.07528/) (2021)
- [Precise Object Placement with Pose Distance Estimations for Different Objects and Grippers](https://mlsanity.com/paper/2110.00992/) (2021)
- [Evaluating Gaussian Grasp Maps for Generative Grasping Models](https://mlsanity.com/paper/2206.00432/) (2022)

## Abstract (arXiv)

In this paper, we introduce a novel learning-based approach for grasping known rigid objects in highly cluttered scenes and precisely placing them based on depth images. Our Placement Quality Network (PQ-Net) estimates the object pose and the quality for each automatically generated grasp pose for multiple objects simultaneously at 92 fps in a single forward pass of a neural network. All grasping and placement trials are executed in a physics simulation and the gained experience is transferred to the real world using domain randomization. We demonstrate that our policy successfully transfers to the real world. PQ-Net outperforms other model-free approaches in terms of grasping success rate and automatically scales to new objects of arbitrary symmetry without any human intervention.