# DemoGrasp: Few-Shot Learning for Robotic Grasping with Human Demonstration

arXiv: https://arxiv.org/abs/2112.02849 · HTML: https://mlsanity.com/paper/2112.02849/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 65 references that are in the map.
- First posted on arXiv 6 Dec 2021.

## Builds on (most cited first, 30 of 37)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/1711.00199/) (2017)
- [QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation](https://mlsanity.com/paper/1806.10293/) (2018)
- [PVNet: Pixel-wise Voting Network for 6DoF Pose Estimation](https://mlsanity.com/paper/1812.11788/) (2018)
- [Learning joint reconstruction of hands and manipulated objects](https://mlsanity.com/paper/1904.05767/) (2019)
- [DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusion](https://mlsanity.com/paper/1901.04780/) (2019)
- [Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection](https://mlsanity.com/paper/1603.02199/) (2016)
- [CosyPose: Consistent multi-view multi-object 6D pose estimation](https://mlsanity.com/paper/2008.08465/) (2020)
- [DeepIM: Deep Iterative Matching for 6D Pose Estimation](https://mlsanity.com/paper/1804.00175/) (2018)
- [Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics](https://mlsanity.com/paper/1703.09312/) (2017)
- [BOP: Benchmark for 6D Object Pose Estimation](https://mlsanity.com/paper/1808.08319/) (2018)
- [SSD-6D: Making RGB-based 3D detection and 6D pose estimation great again](https://mlsanity.com/paper/1711.10006/) (2017)
- [6-DOF GraspNet: Variational Grasp Generation for Object Manipulation](https://mlsanity.com/paper/1905.10520/) (2019)
- [BB8: A Scalable, Accurate, Robust to Partial Occlusion Method for Predicting the 3D Poses of Challenging Objects without Using Depth](https://mlsanity.com/paper/1703.10896/) (2017)
- [Real-Time Seamless Single Shot 6D Object Pose Prediction](https://mlsanity.com/paper/1711.08848/) (2017)
- [Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hours](https://mlsanity.com/paper/1509.06825/) (2015)
- [Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via Randomized-to-Canonical Adaptation Networks](https://mlsanity.com/paper/1812.07252/) (2018)
- [Implicit 3D Orientation Learning for 6D Object Detection from RGB Images](https://mlsanity.com/paper/1902.01275/) (2019)
- [One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learning](https://mlsanity.com/paper/1802.01557/) (2018)
- [EPOS: Estimating 6D Pose of Objects with Symmetries](https://mlsanity.com/paper/2004.00605/) (2020)
- [PPF-FoldNet: Unsupervised Learning of Rotation Invariant 3D Local Descriptors](https://mlsanity.com/paper/1808.10322/) (2018)
- [Closing the Loop for Robotic Grasping: A Real-time, Generative Grasp Synthesis Approach](https://mlsanity.com/paper/1804.05172/) (2018)
- [Deep Learning for Detecting Robotic Grasps](https://mlsanity.com/paper/1301.3592/) (2013)
- [Grasping in the Wild:Learning 6DoF Closed-Loop Grasping from Low-Cost Demonstrations](https://mlsanity.com/paper/1912.04344/) (2019)
- [Self-supervised 6D Object Pose Estimation for Robot Manipulation](https://mlsanity.com/paper/1909.10159/) (2019)
- [Self6D: Self-Supervised Monocular 6D Object Pose Estimation](https://mlsanity.com/paper/2004.06468/) (2020)
- [Real-Time Grasp Detection Using Convolutional Neural Networks](https://mlsanity.com/paper/1412.3128/) (2014)
- [Photorealistic Image Synthesis for Object Instance Detection](https://mlsanity.com/paper/1902.03334/) (2019)

## Built on by (most cited first, 20 of 20)

- [HouseCat6D -- A Large-Scale Multi-Modal Category Level 6D Object Perception Dataset with Household Objects in Realistic Scenarios](https://mlsanity.com/paper/2212.10428/) (2022)
- [In-Hand 3D Object Scanning from an RGB Sequence](https://mlsanity.com/paper/2211.16193/) (2022)
- [On the Importance of Accurate Geometry Data for Dense 3D Vision Tasks](https://mlsanity.com/paper/2303.14840/) (2023)
- [TexPose: Neural Texture Learning for Self-Supervised 6D Object Pose Estimation](https://mlsanity.com/paper/2212.12902/) (2022)
- [Is my Depth Ground-Truth Good Enough? HAMMER -- Highly Accurate Multi-Modal Dataset for DEnse 3D Scene Regression](https://mlsanity.com/paper/2205.04565/) (2022)
- [Meta-Learning Regrasping Strategies for Physical-Agnostic Objects](https://mlsanity.com/paper/2205.11110/) (2022)
- [Quality Diversity under Sparse Reward and Sparse Interaction: Application to Grasping in Robotics](https://mlsanity.com/paper/2308.05483/) (2023)
- [Polaris: Open-ended Interactive Robotic Manipulation via Syn2Real Visual Grounding and Large Language Models](https://mlsanity.com/paper/2408.07975/) (2024)
- [Speeding up 6-DoF Grasp Sampling with Quality-Diversity](https://mlsanity.com/paper/2403.06173/) (2024)
- [NeRF-Feat: 6D Object Pose Estimation using Feature Rendering](https://mlsanity.com/paper/2406.13796/) (2024)
- [Time-to-Label: Temporal Consistency for Self-Supervised Monocular 3D Object Detection](https://mlsanity.com/paper/2203.02193/) (2022)
- [HGDiffuser: Efficient Task-Oriented Grasp Generation via Human-Guided Grasp Diffusion Models](https://mlsanity.com/paper/2503.00508/) (2025)
- [Toward a Plug-and-Play Vision-Based Grasping Module for Robotics](https://mlsanity.com/paper/2310.04349/) (2023)
- [S2P3: Self-Supervised Polarimetric Pose Prediction](https://mlsanity.com/paper/2312.01105/) (2023)
- [QDGset: A Large Scale Grasping Dataset Generated with Quality-Diversity](https://mlsanity.com/paper/2410.02319/) (2024)
- [Remote Task-oriented Grasp Area Teaching By Non-Experts through Interactive Segmentation and Few-Shot Learning](https://mlsanity.com/paper/2303.10195/) (2023)
- [Robotic Learning the Sequence of Packing Irregular Objects from Human Demonstrations](https://mlsanity.com/paper/2210.01645/) (2022)
- [The impact of Compositionality in Zero-shot Multi-label action recognition for Object-based tasks](https://mlsanity.com/paper/2405.08695/) (2024)
- [PickScan: Object discovery and reconstruction from handheld interactions](https://mlsanity.com/paper/2411.11196/) (2024)
- [Paparazzo: Active Mapping of Moving 3D Objects](https://mlsanity.com/paper/2604.19556/) (2026)

## Abstract (arXiv)

The ability to successfully grasp objects is crucial in robotics, as it enables several interactive downstream applications. To this end, most approaches either compute the full 6D pose for the object of interest or learn to predict a set of grasping points. While the former approaches do not scale well to multiple object instances or classes yet, the latter require large annotated datasets and are hampered by their poor generalization capabilities to new geometries. To overcome these shortcomings, we propose to teach a robot how to grasp an object with a simple and short human demonstration. Hence, our approach neither requires many annotated images nor is it restricted to a specific geometry. We first present a small sequence of RGB-D images displaying a human-object interaction. This sequence is then leveraged to build associated hand and object meshes that represent the depicted interaction. Subsequently, we complete missing parts of the reconstructed object shape and estimate the relative transformation between the reconstruction and the visible object in the scene. Finally, we transfer the a-priori knowledge from the relative pose between object and human hand with the estimate of the current object pose in the scene into necessary grasping instructions for the robot. Exhaustive evaluations with Toyota's Human Support Robot (HSR) in real and synthetic environments demonstrate the applicability of our proposed methodology and its advantage in comparison to previous approaches.