mlsanityData to 7 Oct 2026

Learning Generalizable Dexterous Manipulation from Human Grasp Affordance

arXiv 2204.02320 ↗First posted 5 Apr 2022cs.RO cs.CV cs.LGcited by 46

On the map: Offline RL and imitation learning from data → Robot manipulation imitation learning

Abstract (arXiv)

Dexterous manipulation with a multi-finger hand is one of the most challenging problems in robotics. While recent progress in imitation learning has largely improved the sample efficiency compared to Reinforcement Learning, the learned policy can hardly generalize to manipulate novel objects, given limited expert demonstrations. In this paper, we propose to learn dexterous manipulation using large-scale demonstrations with diverse 3D objects in a category, which are generated from a human grasp affordance model. This generalizes the policy to novel object instances within the same category. To train the policy, we propose a novel imitation learning objective jointly with a geometric representation learning objective using our demonstrations. By experimenting with relocating diverse objects in simulation, we show that our approach outperforms baselines with a large margin when manipulating novel objects. We also ablate the importance on 3D object representation learning for manipulation. We include videos, code, and additional information on the project website - https://kristery.github.io/ILAD/ .

Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 53 of its 77 references that are in the map. First posted on arXiv 5 Apr 2022.

Builds on

53 of 77 references.

Closest ancestors · 6

Same direction, newest first

  1. 2022Dexterous Imitation Made Easy: A Learning-Based Framework for Efficient Dexterous Manipulation24 Mar
  2. 2021DexMV: Imitation Learning for Dexterous Manipulation from Human Videos12 Aug
  3. 2020Visual Imitation Made Easy11 Aug
  4. 2020State-Only Imitation Learning for Dexterous Manipulation7 Apr
  5. 2020A Geometric Perspective on Visual Imitation Learning5 Mar
  6. 2019Motion Reasoning for Goal-Based Imitation Learning13 Nov

Other references · 47

  1. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  2. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  3. 2015Trust Region Policy Optimizationcited by 2,503
  4. 2016Generative Adversarial Imitation Learningcited by 1,576
  5. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  6. 2019Solving Rubik's Cube with a Robot Handcited by 699
  7. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  8. 2018Learning Dexterous In-Hand Manipulationcited by 643
  9. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  10. 2018Behavioral Cloning from Observationcited by 482
  11. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 400
  12. 2020GRAB: A Dataset of Whole-Body Human Grasping of Objectscited by 397
35 more
  1. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 361
  2. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 309
  3. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
  4. 2013Challenges in Representation Learning: A report on three machine learning contestscited by 301
  5. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 268
  6. 2020Decoupling Representation Learning from Reinforcement Learningcited by 263
  7. 2018HG-DAgger: Interactive Imitation Learning with Human Expertscited by 245
  8. 2019Deep Dynamics Models for Learning Dexterous Manipulationcited by 200
  9. 2021ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrationscited by 193
  10. 2018Generative Adversarial Imitation from Observationcited by 183
  11. 2020Grasping Field: Learning Implicit Representations for Human Graspscited by 178
  12. 2021Hand-Object Contact Consistency Reasoning for Human Grasps Generationcited by 160
  13. 2021A System for General In-Hand Object Re-Orientationcited by 155
  14. 2019ContactDB: Analyzing and Predicting Grasp Contact via Thermal Imagingcited by 145
  15. 2018Self-Imitation Learningcited by 135
  16. 2020Generalization in Reinforcement Learning by Soft Data Augmentationcited by 132
  17. 2020Self-Supervised Policy Adaptation during Deploymentcited by 121
  18. 2019Imitation Learning from Imperfect Demonstrationcited by 112
  19. 2019DexPilot: Vision Based Teleoperation of Dexterous Robotic Hand-Arm Systemcited by 105
  20. 2019Grasping in the Wild:Learning 6DoF Closed-Loop Grasping from Low-Cost Demonstrationscited by 104
  21. 2017Third-Person Imitation Learningcited by 103
  22. 2018Playing hard exploration games by watching YouTubecited by 98
  23. 2019ContactGrasp: Functional Multi-finger Grasp Synthesis from Contactcited by 87
  24. 2020Learning Dexterous Grasping with Object-Centric Visual Affordancescited by 80
  25. 2020Reinforcement Learning with Videos: Combining Offline Observations with Interactioncited by 80
  26. 2018Dexterous Manipulation with Deep Reinforcement Learning: Efficient, General, and Low-Costcited by 79
  27. 2022Robotic Telekinesis: Learning a Robotic Hand Imitator by Watching Humans on Youtubecited by 79
  28. 2019State Alignment-based Imitation Learningcited by 73
  29. 2021Synthesizing Diverse and Physically Stable Grasps with Arbitrary Hand Structures using Differentiable Force Closure Estimatorcited by 70
  30. 2021Generalization in Dexterous Manipulation via Geometry-Aware Multi-Task Learningcited by 49
  31. 2016Learning Dexterous Manipulation for a Soft Robotic Hand from Human Demonstrationcited by 35
  32. 2016Learning Dexterous Manipulation Policies from Experience and Imitationcited by 32
  33. 2021Learning from Imperfect Demonstrations from Agents with Varying Dynamicscited by 26
  34. 2018Planning Multi-Fingered Grasps as Probabilistic Inference in a Learned Deep Networkcited by 19
  35. 2019VILD: Variational Imitation Learning with Diverse-quality Demonstrationscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 14; 2024: 12; 2025: 11; 2026: 3.

46 papers.

Sort:

By area:

  1. 2024RoboMIND: Benchmark on Multi-embodiment Intelligence Normative Data for Robot Manipulationcited by 166
  2. 2023UniDexGrasp: Universal Robotic Dexterous Grasping via Learning Diverse Proposal Generation and Goal-Conditioned Policycited by 117
  3. 2023Affordance Diffusion: Synthesizing Hand-Object Interactionscited by 110
  4. 2023UniDexGrasp++: Improving Dexterous Grasping Policy Learning via Geometry-aware Curriculum and Iterative Generalist-Specialist Learningcited by 101
  5. 2024Move as You Say, Interact as You Can: Language-guided Human Motion Generation with Scene Affordancecited by 79
  6. 2022DexPoint: Generalizable Point Cloud Reinforcement Learning for Sim-to-Real Dexterous Manipulationcited by 78
  7. 2023Rotating without Seeing: Towards In-hand Dexterity through Touchcited by 78
  8. 2024Robo-ABC: Affordance Generalization Beyond Categories via Semantic Correspondence for Robot Manipulationcited by 69
  9. 2023Elastic Decision Transformercited by 66
  10. 2023DexArt: Benchmarking Generalizable Dexterous Manipulation with Articulated Objectscited by 64
  11. 2022End-to-End Affordance Learning for Robotic Manipulationcited by 54
  12. 2023Robot Synesthesia: In-Hand Manipulation with Visuotactile Sensingcited by 51
34 more
  1. 2023PartManip: Learning Cross-Category Generalizable Part Manipulation Policy from Point Cloud Observationscited by 40
  2. 2022Learning Continuous Grasping Function with a Dexterous Hand from Human Demonstrationscited by 34
  3. 2024RealDex: Towards Human-like Grasping for Robotic Dexterous Handcited by 33
  4. 2024ViViDex: Learning Vision-based Dexterous Manipulation from Human Videoscited by 33
  5. 2025GraspGen: A Diffusion-based Framework for 6-DOF Grasping with On-Generator Trainingcited by 32
  6. 2022Frame Mining: a Free Lunch for Learning Robotic Manipulation from 3D Point Cloudscited by 31
  7. 2023GraspGF: Learning Score-based Grasping Primitive for Human-assisting Dexterous Graspingcited by 23
  8. 2025UAD: Unsupervised Affordance Distillation for Generalization in Robotic Manipulationcited by 23
  9. 2023SparseDFF: Sparse-View Feature Distillation for One-Shot Dexterous Manipulationcited by 22
  10. 2024PhysReaction: Physically Plausible Real-Time Humanoid Reaction Synthesis via Forward Dynamics Guided 4D Imitationcited by 18
  11. 2023A Universal Semantic-Geometric Representation for Robotic Manipulationcited by 16
  12. 2023SynH2R: Synthesizing Hand-Object Motions for Learning Human-to-Robot Handoverscited by 13
  13. 2024Leveraging Locality to Boost Sample Efficiency in Robotic Manipulationcited by 13
  14. 2025DexTrack: Towards Generalizable Neural Tracking Control for Dexterous Manipulation from Human Referencescited by 13
  15. 20233D Reconstruction of Objects in Hands without Real World 3D Supervisioncited by 12
  16. 2024ManiFoundation Model for General-Purpose Robotic Manipulation of Contact Synthesis with Arbitrary Objects and Robotscited by 11
  17. 2024QuasiSim: Parameterized Quasi-Physical Simulators for Dexterous Manipulations Transfercited by 10
  18. 2023On the Efficacy of 3D Point Cloud Reinforcement Learningcited by 9
  19. 2022HERD: Continuous Human-to-Robot Evolution for Learning from Human Demonstrationcited by 8
  20. 2025DyTact: Capturing Dynamic Contacts in Hand-Object Manipulationcited by 8
  21. 2025DexMan: Learning Bimanual Dexterous Manipulation from Human and Generated Videoscited by 7
  22. 2024PointPatchRL -- Masked Reconstruction Improves Reinforcement Learning on Point Cloudscited by 5
  23. 2024Neural Attention Field: Emerging Point Relevance in 3D Scenes for One-Shot Dexterous Graspingcited by 5
  24. 2022Efficient Representations of Object Geometry for Reinforcement Learning of Interactive Grasping Policiescited by 2
  25. 2024Dexterous Manipulation Based on Prior Dexterous Grasp Pose Knowledgecited by 2
  26. 2025VTAO-BiManip: Masked Visual-Tactile-Action Pre-training with Object Understanding for Bimanual Dexterous Manipulationcited by 2
  27. 2025FunHOI: Annotation-Free 3D Hand-Object Interaction Generation via Functional Text Guidancecited by 2
  28. 2025Mash, Spread, Slice! Learning to Manipulate Object States via Visual Spatial Progresscited by 2
  29. 2026VideoAfford: Grounding 3D Affordance from Human-Object-Interaction Videos via Multimodal Large Language Modelcited by 2
  30. 2025Task-Oriented Human Grasp Synthesis via Context- and Task-Aware Diffuserscited by 1
  31. 2025Robust Point Cloud Reinforcement Learning via PCA-Based Canonicalizationcited by 1
  32. 2025Target Return Optimizer for Multi-Game Decision Transformercited by 0
  33. 2026ContactExplorer: Contact Coverage-Guided Exploration for General-Purpose Dexterous Manipulationcited by 0
  34. 2026Tac2Pix: Image-Space Visuo-Tactile Fusion for Dexterous Manipulationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Dexterous Manipulation through Imitation Learning: A Survey
  2. 2024Dexterous Functional Pre-Grasp Manipulation with Diffusion Policy
  3. 2026From Grasps to Dexterity: Large-Scale Grasp Pretraining for Dexterous Manipulation
  4. 2022From One Hand to Multiple Hands: Imitation Learning for Dexterous Manipulation from Single-Camera Teleoperation
  5. 2022Learning Category-Level Generalizable Object Manipulation Policy via Generative Adversarial Self-Imitation Learning from Demonstrations
  6. 2026FSAG: Enhancing Human-to-Dexterous-Hand Finger-Specific Affordance Grounding via Diffusion Models
  7. 2025Learning Generalizable Hand-Object Tracking from Synthetic Demonstrations
  8. 2023Learning to Transfer In-Hand Manipulations Using a Greedy Shape Curriculum