# KGNv2: Separating Scale and Pose Prediction for Keypoint-based 6-DoF Grasp Synthesis on RGB-D input

arXiv: https://arxiv.org/abs/2303.05617 · HTML: https://mlsanity.com/paper/2303.05617/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 53 references that are in the map.
- First posted on arXiv 9 Mar 2023.

## Builds on (most cited first, 30 of 37)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [Objects as Points](https://mlsanity.com/paper/1904.07850/) (2019)
- [CenterNet: Keypoint Triplets for Object Detection](https://mlsanity.com/paper/1904.08189/) (2019)
- [Deep Layer Aggregation](https://mlsanity.com/paper/1707.06484/) (2017)
- [TEXTure: Text-Guided Texturing of 3D Shapes](https://mlsanity.com/paper/2302.01721/) (2023)
- [Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics](https://mlsanity.com/paper/1703.09312/) (2017)
- [6-DOF GraspNet: Variational Grasp Generation for Object Manipulation](https://mlsanity.com/paper/1905.10520/) (2019)
- [Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenes](https://mlsanity.com/paper/2103.14127/) (2021)
- [ACRONYM: A Large-Scale Grasp Dataset Based on Simulation](https://mlsanity.com/paper/2011.09584/) (2020)
- [Bottom-Up Human Pose Estimation Via Disentangled Keypoint Regression](https://mlsanity.com/paper/2104.02300/) (2021)
- [Grasp Pose Detection in Point Clouds](https://mlsanity.com/paper/1706.09911/) (2017)
- [Jacquard: A Large Scale Dataset for Robotic Grasp Detection](https://mlsanity.com/paper/1803.11469/) (2018)
- [PointNetGPD: Detecting Grasp Configurations from Point Sets](https://mlsanity.com/paper/1809.06267/) (2018)
- [Deep Learning Approaches to Grasp Synthesis: A Review](https://mlsanity.com/paper/2207.02556/) (2022)
- [YOLO-Pose: Enhancing YOLO for Multi Person Pose Estimation Using Object Keypoint Similarity Loss](https://mlsanity.com/paper/2204.06806/) (2022)
- [S4G: Amodal Single-view Single-Shot SE(3) Grasp Detection in Cluttered Scenes](https://mlsanity.com/paper/1910.14218/) (2019)
- [Single-Stage Keypoint-Based Category-Level Object Pose Estimation from an RGB Image](https://mlsanity.com/paper/2109.06161/) (2021)
- [PointNet++ Grasping: Learning An End-to-end Spatial Grasp Generation Algorithm from Sparse Point Clouds](https://mlsanity.com/paper/2003.09644/) (2020)
- [RGB Matters: Learning 7-DoF Grasp Poses on Monocular RGBD Images](https://mlsanity.com/paper/2103.02184/) (2021)
- [A Billion Ways to Grasp: An Evaluation of Grasp Sampling Schemes on a Dense, Physics-based Grasp Data Set](https://mlsanity.com/paper/1912.05604/) (2019)
- [MonoGraspNet: 6-DoF Grasping with a Single RGB Image](https://mlsanity.com/paper/2209.13036/) (2022)
- [A Joint Network for Grasp Detection Conditioned on Natural Language Commands](https://mlsanity.com/paper/2104.00492/) (2021)
- [Grasp Proposal Networks: An End-to-End Solution for Visual Learning of Robotic Grasps](https://mlsanity.com/paper/2009.12606/) (2020)
- [Towards Scale Balanced 6-DoF Grasp Detection in Cluttered Scenes](https://mlsanity.com/paper/2212.05275/) (2022)
- [Planning Multi-Fingered Grasps as Probabilistic Inference in a Learned Deep Network](https://mlsanity.com/paper/1804.03289/) (2018)

## Built on by (most cited first, 4 of 4)

- [Learning Any-View 6DoF Robotic Grasping in Cluttered Scenes via Neural Surface Rendering](https://mlsanity.com/paper/2306.07392/) (2023)
- [Constrained 6-DoF Grasp Generation on Complex Shapes for Improved Dual-Arm Manipulation](https://mlsanity.com/paper/2404.04643/) (2024)
- [Real-time Simultaneous Multi-Object 3D Shape Reconstruction, 6DoF Pose Estimation and Dense Grasp Prediction](https://mlsanity.com/paper/2305.09510/) (2023)
- [SpikeGrasp: A Benchmark for 6-DoF Grasp Pose Detection from Stereo Spike Streams](https://mlsanity.com/paper/2510.10602/) (2025)

## Abstract (arXiv)

We propose a new 6-DoF grasp pose synthesis approach from 2D/2.5D input based on keypoints. Keypoint-based grasp detector from image input has demonstrated promising results in the previous study, where the additional visual information provided by color images compensates for the noisy depth perception. However, it relies heavily on accurately predicting the location of keypoints in the image space. In this paper, we devise a new grasp generation network that reduces the dependency on precise keypoint estimation. Given an RGB-D input, our network estimates both the grasp pose from keypoint detection as well as scale towards the camera. We further re-design the keypoint output space in order to mitigate the negative impact of keypoint prediction noise to Perspective-n-Point (PnP) algorithm. Experiments show that the proposed method outperforms the baseline by a large margin, validating the efficacy of our approach. Finally, despite trained on simple synthetic objects, our method demonstrate sim-to-real capacity by showing competitive results in real-world robot experiments.