# Depth-aware Object Segmentation and Grasp Detection for Robotic Picking Tasks

arXiv: https://arxiv.org/abs/2111.11114 · HTML: https://mlsanity.com/paper/2111.11114/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 44 references that are in the map.
- First posted on arXiv 22 Nov 2021.

## Builds on (most cited first, 30 of 32)

- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation](https://mlsanity.com/paper/1806.10293/) (2018)
- [What Makes Training Multi-Modal Classification Networks Hard?](https://mlsanity.com/paper/1905.12681/) (2019)
- [An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solution](https://mlsanity.com/paper/1807.03247/) (2018)
- [6-DOF GraspNet: Variational Grasp Generation for Object Manipulation](https://mlsanity.com/paper/1905.10520/) (2019)
- [Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hours](https://mlsanity.com/paper/1509.06825/) (2015)
- [Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via Randomized-to-Canonical Adaptation Networks](https://mlsanity.com/paper/1812.07252/) (2018)
- [Learning Rich Features from RGB-D Images for Object Detection and Segmentation](https://mlsanity.com/paper/1407.5736/) (2014)
- [Deep Learning for Detecting Robotic Grasps](https://mlsanity.com/paper/1301.3592/) (2013)
- [Seamless Scene Segmentation](https://mlsanity.com/paper/1905.01220/) (2019)
- [Data-Driven Grasp Synthesis - A Survey](https://mlsanity.com/paper/1309.2660/) (2013)
- [Unseen Object Instance Segmentation for Robotic Environments](https://mlsanity.com/paper/2007.08073/) (2020)
- [Real-Time Grasp Detection Using Convolutional Neural Networks](https://mlsanity.com/paper/1412.3128/) (2014)
- [Depth-aware CNN for RGB-D Segmentation](https://mlsanity.com/paper/1803.06791/) (2018)
- [6-DOF Grasping for Target-driven Object Manipulation in Clutter](https://mlsanity.com/paper/1912.03628/) (2019)
- [AdaptIS: Adaptive Instance Selection Network](https://mlsanity.com/paper/1909.07829/) (2019)
- [EasyLabel: A Semi-Automatic Pixel-wise Object Annotation Tool for Creating Robotic RGB-D Datasets](https://mlsanity.com/paper/1902.01626/) (2019)
- [Robotic Grasp Detection using Deep Convolutional Neural Networks](https://mlsanity.com/paper/1611.08036/) (2016)
- [End-to-end Trainable Deep Neural Network for Robotic Grasp Detection and Semantic Segmentation from RGB](https://mlsanity.com/paper/2107.05287/) (2021)
- [Deep Reinforcement Learning for Vision-Based Robotic Grasping: A Simulated Comparative Evaluation of Off-Policy Methods](https://mlsanity.com/paper/1802.10264/) (2018)
- [Fully Convolutional Grasp Detection Network with Oriented Anchor Box](https://mlsanity.com/paper/1803.02209/) (2018)
- [Symmetry Aware Evaluation of 3D Object Detection and Pose Estimation in Scenes of Many Parts in Bulk](https://mlsanity.com/paper/1806.08129/) (2018)
- [Densely Supervised Grasp Detector (DSGD)](https://mlsanity.com/paper/1810.03962/) (2018)
- [A Single Multi-Task Deep Neural Network with Post-Processing for Object Detection with Reasoning and Robotic Grasp Detection](https://mlsanity.com/paper/1909.07050/) (2019)

## Built on by (most cited first, 3 of 3)

- [H3DP: Triply-Hierarchical Diffusion Policy for Visuomotor Learning](https://mlsanity.com/paper/2505.07819/) (2025)
- [Sim-Suction: Learning a Suction Grasp Policy for Cluttered Environments Using a Synthetic Benchmark](https://mlsanity.com/paper/2305.16378/) (2023)
- [3D-Aware Multi-Task Learning with Cross-View Correlations for Dense Scene Understanding](https://mlsanity.com/paper/2511.20646/) (2025)

## Abstract (arXiv)

In this paper, we present a novel deep neural network architecture for joint class-agnostic object segmentation and grasp detection for robotic picking tasks using a parallel-plate gripper. We introduce depth-aware Coordinate Convolution (CoordConv), a method to increase accuracy for point proposal based object instance segmentation in complex scenes without adding any additional network parameters or computation complexity. Depth-aware CoordConv uses depth data to extract prior information about the location of an object to achieve highly accurate object instance segmentation. These resulting segmentation masks, combined with predicted grasp candidates, lead to a complete scene description for grasping using a parallel-plate gripper. We evaluate the accuracy of grasp detection and instance segmentation on challenging robotic picking datasets, namely Sil'eane and OCIDgrasp, and show the benefit of joint grasp detection and segmentation on a real-world robotic picking task.