# Keypoint-GraspNet: Keypoint-based 6-DoF Grasp Generation from the Monocular RGB-D input

arXiv: https://arxiv.org/abs/2209.08752 · HTML: https://mlsanity.com/paper/2209.08752/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 48 references that are in the map.
- First posted on arXiv 19 Sep 2022.

## Builds on (most cited first, 30 of 31)

- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [Objects as Points](https://mlsanity.com/paper/1904.07850/) (2019)
- [CenterNet: Keypoint Triplets for Object Detection](https://mlsanity.com/paper/1904.08189/) (2019)
- [RandLA-Net: Efficient Semantic Segmentation of Large-Scale Point Clouds](https://mlsanity.com/paper/1911.11236/) (2019)
- [Deep Layer Aggregation](https://mlsanity.com/paper/1707.06484/) (2017)
- [Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics](https://mlsanity.com/paper/1703.09312/) (2017)
- [6-DOF GraspNet: Variational Grasp Generation for Object Manipulation](https://mlsanity.com/paper/1905.10520/) (2019)
- [Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenes](https://mlsanity.com/paper/2103.14127/) (2021)
- [ACRONYM: A Large-Scale Grasp Dataset Based on Simulation](https://mlsanity.com/paper/2011.09584/) (2020)
- [Grasp Pose Detection in Point Clouds](https://mlsanity.com/paper/1706.09911/) (2017)
- [Volumetric Grasping Network: Real-time 6 DOF Grasp Detection in Clutter](https://mlsanity.com/paper/2101.01132/) (2021)
- [Jacquard: A Large Scale Dataset for Robotic Grasp Detection](https://mlsanity.com/paper/1803.11469/) (2018)
- [PointNetGPD: Detecting Grasp Configurations from Point Sets](https://mlsanity.com/paper/1809.06267/) (2018)
- [S4G: Amodal Single-view Single-Shot SE(3) Grasp Detection in Cluttered Scenes](https://mlsanity.com/paper/1910.14218/) (2019)
- [Object Rearrangement Using Learned Implicit Collision Functions](https://mlsanity.com/paper/2011.10726/) (2020)
- [Single-Stage Keypoint-Based Category-Level Object Pose Estimation from an RGB Image](https://mlsanity.com/paper/2109.06161/) (2021)
- [PointNet++ Grasping: Learning An End-to-end Spatial Grasp Generation Algorithm from Sparse Point Clouds](https://mlsanity.com/paper/2003.09644/) (2020)
- [RGB Matters: Learning 7-DoF Grasp Poses on Monocular RGBD Images](https://mlsanity.com/paper/2103.02184/) (2021)
- [A Billion Ways to Grasp: An Evaluation of Grasp Sampling Schemes on a Dense, Physics-based Grasp Data Set](https://mlsanity.com/paper/1912.05604/) (2019)
- [Reactive Human-to-Robot Handovers of Arbitrary Objects](https://mlsanity.com/paper/2011.08961/) (2020)
- [A Joint Network for Grasp Detection Conditioned on Natural Language Commands](https://mlsanity.com/paper/2104.00492/) (2021)
- [Grasp Proposal Networks: An End-to-End Solution for Visual Learning of Robotic Grasps](https://mlsanity.com/paper/2009.12606/) (2020)
- [Planning Multi-Fingered Grasps as Probabilistic Inference in a Learned Deep Network](https://mlsanity.com/paper/1804.03289/) (2018)
- [PerspectiveNet: 3D Object Detection from a Single RGB Image via Perspective Points](https://mlsanity.com/paper/1912.07744/) (2019)
- [Collision-Aware Target-Driven Object Grasping in Constrained Environments](https://mlsanity.com/paper/2104.00776/) (2021)
- [GDN: A Coarse-To-Fine (C2F) Representation for End-To-End 6-DoF Grasp Detection](https://mlsanity.com/paper/2010.10695/) (2020)
- [GKNet: grasp keypoint network for grasp candidates detection](https://mlsanity.com/paper/2106.08497/) (2021)

## Built on by (most cited first, 10 of 10)

- [HouseCat6D -- A Large-Scale Multi-Modal Category Level 6D Object Perception Dataset with Household Objects in Realistic Scenarios](https://mlsanity.com/paper/2212.10428/) (2022)
- [Lightweight Language-driven Grasp Detection using Conditional Consistency Model](https://mlsanity.com/paper/2407.17967/) (2024)
- [Learning Any-View 6DoF Robotic Grasping in Cluttered Scenes via Neural Surface Rendering](https://mlsanity.com/paper/2306.07392/) (2023)
- [KGNv2: Separating Scale and Pose Prediction for Keypoint-based 6-DoF Grasp Synthesis on RGB-D input](https://mlsanity.com/paper/2303.05617/) (2023)
- [Real-time Simultaneous Multi-Object 3D Shape Reconstruction, 6DoF Pose Estimation and Dense Grasp Prediction](https://mlsanity.com/paper/2305.09510/) (2023)
- [Grammarization-Based Grasping with Deep Multi-Autoencoder Latent Space Exploration by Reinforcement Learning Agent](https://mlsanity.com/paper/2411.08566/) (2024)
- [Robust Loss Functions for Object Grasping under Limited Ground Truth](https://mlsanity.com/paper/2409.05742/) (2024)
- [GAA-TSO: Geometry-Aware Assisted Depth Completion for Transparent and Specular Objects](https://mlsanity.com/paper/2503.17106/) (2025)
- [ISS Policy : Scalable Diffusion Policy with Implicit Scene Supervision](https://mlsanity.com/paper/2512.15020/) (2025)
- [Enabling a Unified Cross-Domain Representation for Two-Finger Gripper Manipulation via Interaction-Centric Modeling](https://mlsanity.com/paper/2609.31207/) (2026)

## Abstract (arXiv)

Great success has been achieved in the 6-DoF grasp learning from the point cloud input, yet the computational cost due to the point set orderlessness remains a concern. Alternatively, we explore the grasp generation from the RGB-D input in this paper. The proposed solution, Keypoint-GraspNet, detects the projection of the gripper keypoints in the image space and then recover the SE(3) poses with a PnP algorithm. A synthetic dataset based on the primitive shape and the grasp family is constructed to examine our idea. Metric-based evaluation reveals that our method outperforms the baselines in terms of the grasp proposal accuracy, diversity, and the time cost. Finally, robot experiments show high success rate, demonstrating the potential of the idea in the real-world applications.