# Simultaneous Semantic and Collision Learning for 6-DoF Grasp Pose Estimation

arXiv: https://arxiv.org/abs/2108.02425 · HTML: https://mlsanity.com/paper/2108.02425/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 40 references that are in the map.
- First posted on arXiv 5 Aug 2021.

## Builds on (most cited first, 30 of 31)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud](https://mlsanity.com/paper/1812.04244/) (2018)
- [PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/1711.00199/) (2017)
- [3D Semantic Segmentation with Submanifold Sparse Convolutional Networks](https://mlsanity.com/paper/1711.10275/) (2017)
- [DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusion](https://mlsanity.com/paper/1901.04780/) (2019)
- [PointGroup: Dual-Set Point Grouping for 3D Instance Segmentation](https://mlsanity.com/paper/2004.01658/) (2020)
- [Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection](https://mlsanity.com/paper/1603.02199/) (2016)
- [Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics](https://mlsanity.com/paper/1703.09312/) (2017)
- [6-DOF GraspNet: Variational Grasp Generation for Object Manipulation](https://mlsanity.com/paper/1905.10520/) (2019)
- [Volumetric and Multi-View CNNs for Object Classification on 3D Data](https://mlsanity.com/paper/1604.03265/) (2016)
- [Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hours](https://mlsanity.com/paper/1509.06825/) (2015)
- [Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputs](https://mlsanity.com/paper/1703.09438/) (2017)
- [Closing the Loop for Robotic Grasping: A Real-time, Generative Grasp Synthesis Approach](https://mlsanity.com/paper/1804.05172/) (2018)
- [Semantic Instance Segmentation with a Discriminative Loss Function](https://mlsanity.com/paper/1708.02551/) (2017)
- [ACRONYM: A Large-Scale Grasp Dataset Based on Simulation](https://mlsanity.com/paper/2011.09584/) (2020)
- [Deep Learning for Detecting Robotic Grasps](https://mlsanity.com/paper/1301.3592/) (2013)
- [Grasp Pose Detection in Point Clouds](https://mlsanity.com/paper/1706.09911/) (2017)
- [Data-Driven Grasp Synthesis - A Survey](https://mlsanity.com/paper/1309.2660/) (2013)
- [Self-supervised 6D Object Pose Estimation for Robot Manipulation](https://mlsanity.com/paper/1909.10159/) (2019)
- [Jacquard: A Large Scale Dataset for Robotic Grasp Detection](https://mlsanity.com/paper/1803.11469/) (2018)
- [PointNetGPD: Detecting Grasp Configurations from Point Sets](https://mlsanity.com/paper/1809.06267/) (2018)
- [Real-Time Grasp Detection Using Convolutional Neural Networks](https://mlsanity.com/paper/1412.3128/) (2014)
- [6-DOF Grasping for Target-driven Object Manipulation in Clutter](https://mlsanity.com/paper/1912.03628/) (2019)
- [S4G: Amodal Single-view Single-Shot SE(3) Grasp Detection in Cluttered Scenes](https://mlsanity.com/paper/1910.14218/) (2019)
- [Robotic Grasp Detection using Deep Convolutional Neural Networks](https://mlsanity.com/paper/1611.08036/) (2016)
- [PointNet++ Grasping: Learning An End-to-end Spatial Grasp Generation Algorithm from Sparse Point Clouds](https://mlsanity.com/paper/2003.09644/) (2020)
- [High precision grasp pose detection in dense clutter](https://mlsanity.com/paper/1603.01564/) (2016)
- [A Billion Ways to Grasp: An Evaluation of Grasp Sampling Schemes on a Dense, Physics-based Grasp Data Set](https://mlsanity.com/paper/1912.05604/) (2019)

## Built on by (most cited first, 14 of 14)

- [Spherical Transformer for LiDAR-based 3D Recognition](https://mlsanity.com/paper/2303.12766/) (2023)
- [LargeKernel3D: Scaling up Kernels in 3D Sparse CNNs](https://mlsanity.com/paper/2206.10555/) (2022)
- [IST-Net: Prior-free Category-level Pose Estimation with Implicit Space Transformation](https://mlsanity.com/paper/2303.13479/) (2023)
- [SUGAR: Pre-training 3D Visual Representations for Robotics](https://mlsanity.com/paper/2404.01491/) (2024)
- [An Economic Framework for 6-DoF Grasp Detection](https://mlsanity.com/paper/2407.08366/) (2024)
- [Towards Scale Balanced 6-DoF Grasp Detection in Cluttered Scenes](https://mlsanity.com/paper/2212.05275/) (2022)
- [Deep Fusion Transformer Network with Weighted Vector-Wise Keypoints Voting for Robust 6D Object Pose Estimation](https://mlsanity.com/paper/2308.05438/) (2023)
- [ICGNet: A Unified Approach for Instance-Centric Grasping](https://mlsanity.com/paper/2401.09939/) (2024)
- [Where To Start? Transferring Simple Skills to Complex Environments](https://mlsanity.com/paper/2212.06111/) (2022)
- [Cluttered Food Grasping with Adaptive Fingers and Synthetic-Data Trained Object Detection](https://mlsanity.com/paper/2203.05187/) (2022)
- [GraNet: A Multi-Level Graph Network for 6-DoF Grasp Pose Generation in Cluttered Scenes](https://mlsanity.com/paper/2312.03345/) (2023)
- [You Only Scan Once: A Dynamic Scene Reconstruction Pipeline for 6-DoF Robotic Grasping of Novel Objects](https://mlsanity.com/paper/2404.03462/) (2024)
- [SKU-Patch: Towards Efficient Instance Segmentation for Unseen Objects in Auto-Store](https://mlsanity.com/paper/2311.04645/) (2023)
- [Self-supervised 6-DoF Robot Grasping by Demonstration via Augmented Reality Teleoperation System](https://mlsanity.com/paper/2404.03067/) (2024)

## Abstract (arXiv)

Grasping in cluttered scenes has always been a great challenge for robots, due to the requirement of the ability to well understand the scene and object information. Previous works usually assume that the geometry information of the objects is available, or utilize a step-wise, multi-stage strategy to predict the feasible 6-DoF grasp poses. In this work, we propose to formalize the 6-DoF grasp pose estimation as a simultaneous multi-task learning problem. In a unified framework, we jointly predict the feasible 6-DoF grasp poses, instance semantic segmentation, and collision information. The whole framework is jointly optimized and end-to-end differentiable. Our model is evaluated on large-scale benchmarks as well as the real robot system. On the public dataset, our method outperforms prior state-of-the-art methods by a large margin (+4.08 AP). We also demonstrate the implementation of our model on a real robotic platform and show that the robot can accurately grasp target objects in cluttered scenarios with a high success rate. Project link: https://openbyterobotics.github.io/sscl