mlsanityData to 7 Oct 2026

DexVIP: Learning Dexterous Grasping with Human Hand Pose Priors from Video

arXiv 2202.00164 ↗First posted 1 Feb 2022cs.RO cs.CVcited by 83

Not on the map. Why?

Abstract (arXiv)

Dexterous multi-fingered robotic hands have a formidable action space, yet their morphological similarity to the human hand holds immense potential to accelerate robot learning. We propose DexVIP, an approach to learn dexterous robotic grasping from human-object interactions present in in-the-wild YouTube videos. We do this by curating grasp images from human-object interaction videos and imposing a prior over the agent's hand pose when learning to grasp with deep reinforcement learning. A key advantage of our method is that the learned policy is able to leverage free-form in-the-wild visual data. As a result, it can easily scale to new objects, and it sidesteps the standard practice of collecting human demonstrations in a lab -- a much more expensive and indirect way to capture human expertise. Through experiments on 27 objects with a 30-DoF simulated robot hand, we demonstrate that DexVIP compares favorably to existing approaches that lack a hand pose prior or rely on specialized tele-operation equipment to obtain human demonstrations, while also being faster to train. Project page: https://vision.cs.utexas.edu/projects/dexvip-dexterous-grasp-pose-prior

Key facts. Cited by 83 papers in the map (data through 7 Oct 2026). Builds on 48 of its 59 references that are in the map. First posted on arXiv 1 Feb 2022.

Builds on

48 of 59 references.

References · 48

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  3. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  4. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  5. 2019Solving Rubik's Cube with a Robot Handcited by 699
  6. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  7. 2018Learning Dexterous In-Hand Manipulationcited by 643
  8. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 453
  9. 2020GRAB: A Dataset of Whole-Body Human Grasping of Objectscited by 397
  10. 2019Learning joint reconstruction of hands and manipulated objectscited by 377
  11. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
  12. 2021DexYCB: A Benchmark for Capturing Hand Grasping of Objectscited by 320
36 more
  1. 2016Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collectioncited by 312
  2. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
  3. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  4. 2017Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metricscited by 259
  5. 2017One-Shot Visual Imitation Learning via Meta-Learningcited by 225
  6. 2019Deep Dynamics Models for Learning Dexterous Manipulationcited by 200
  7. 2020ContactPose: A Dataset of Grasps with Object Contact and Hand Posecited by 180
  8. 2017Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translationcited by 167
  9. 2019ContactDB: Analyzing and Predicting Grasp Contact via Thermal Imagingcited by 145
  10. 2020Reconstructing Hand-Object Interactions in the Wildcited by 143
  11. 2018Grounded Human-Object Interaction Hotspots from Videocited by 128
  12. 2017AffordanceNet: An End-to-End Deep Learning Approach for Object Affordance Detectioncited by 126
  13. 2018Reinforcement and Imitation Learning for Diverse Visuomotor Skillscited by 124
  14. 2013Deep Learning for Detecting Robotic Graspscited by 114
  15. 2018Zero-Shot Visual Imitationcited by 114
  16. 2020Monocular Real-time Hand Shape and Motion Capture using Multi-modal Datacited by 106
  17. 2019DexPilot: Vision Based Teleoperation of Dexterous Robotic Hand-Arm Systemcited by 105
  18. 2017Third-Person Imitation Learningcited by 103
  19. 2019Recent Advances in Imitation Learning from Observationcited by 92
  20. 2020State-Only Imitation Learning for Dexterous Manipulationcited by 90
  21. 2019ContactGrasp: Functional Multi-finger Grasp Synthesis from Contactcited by 87
  22. 2018Dexterous Manipulation with Deep Reinforcement Learning: Efficient, General, and Low-Costcited by 79
  23. 2018Multiple Interactions Made Easy (MIME): Large Scale Demonstrations Data for Imitationcited by 76
  24. 2014Real-Time Grasp Detection Using Convolutional Neural Networkscited by 75
  25. 2019Third-Person Visual Imitation Learning via Decoupled Hierarchical Controllercited by 73
  26. 2020FrankMocap: Fast Monocular 3D Hand and Body Motion Capture by Regression and Integrationcited by 72
  27. 2021Learning to See before Learning to Act: Visual Pre-training for Manipulationcited by 50
  28. 2014Weakly Supervised Action Labeling in Videos Under Ordering Constraintscited by 48
  29. 2016Learning Dexterous Manipulation for a Soft Robotic Hand from Human Demonstrationcited by 35
  30. 2018Deep Reinforcement Learning for Vision-Based Robotic Grasping: A Simulated Comparative Evaluation of Off-Policy Methodscited by 32
  31. 2019Learning Task-Oriented Grasping from Human Activity Datasetscited by 32
  32. 2020Grasp Proposal Networks: An End-to-End Solution for Visual Learning of Robotic Graspscited by 22
  33. 2018One-Shot Learning of Multi-Step Tasks from Observation via Activity Localization in Auxiliary Videocited by 16
  34. 2018Leveraging Contact Forces for Learning to Graspcited by 12
  35. 2016Gesture-based Bootstrapping for Egocentric Hand Segmentationcited by 1
  36. 2018Center-of-Mass-Based Grasp Pose Adaptation Using 3D Range and Force/Torque Sensingcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 9; 2023: 12; 2024: 26; 2025: 21; 2026: 15.

83 papers.

Sort:

By area:

  1. 2022Masked Visual Pre-training for Motor Controlcited by 246
  2. 2024RoboMIND: Benchmark on Multi-embodiment Intelligence Normative Data for Robot Manipulationcited by 166
  3. 2024DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulationcited by 149
  4. 2023Structured World Models from Human Videoscited by 144
  5. 2023UniDexGrasp: Universal Robotic Dexterous Grasping via Learning Diverse Proposal Generation and Goal-Conditioned Policycited by 117
  6. 2023Affordance Diffusion: Synthesizing Hand-Object Interactionscited by 110
  7. 2022VideoDex: Learning Dexterity from Internet Videoscited by 101
  8. 2023UniDexGrasp++: Improving Dexterous Grasping Policy Learning via Geometry-aware Curriculum and Iterative Generalist-Specialist Learningcited by 101
  9. 2025DexGraspVLA: A Vision-Language-Action Framework Towards General Dexterous Graspingcited by 87
  10. 2022Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applicationscited by 78
  11. 2023LEAP Hand: Low-Cost, Efficient, and Anthropomorphic Hand for Robot Learningcited by 78
  12. 2022Dexterous Imitation Made Easy: A Learning-Based Framework for Efficient Dexterous Manipulationcited by 75
38 more
  1. 2024Robo-ABC: Affordance Generalization Beyond Categories via Semantic Correspondence for Robot Manipulationcited by 69
  2. 2025Scalable Vision-Language-Action Model Pretraining for Robotic Manipulation with Real-Life Human Activity Videoscited by 63
  3. 2024GraspXL: Generating Grasping Motions for Diverse Objects at Scalecited by 59
  4. 2022GenDexGrasp: Generalizable Dexterous Graspingcited by 54
  5. 2023ArtiGrasp: Physically Plausible Synthesis of Bi-Manual Dexterous Grasping and Articulationcited by 53
  6. 2024HRP: Human Affordances for Robotic Pre-Trainingcited by 40
  7. 2024Hand-Object Interaction Pretraining from Videoscited by 38
  8. 2023Affordance Grounding from Demonstration Video to Target Imagecited by 34
  9. 2024ViViDex: Learning Vision-based Dexterous Manipulation from Human Videoscited by 33
  10. 2024Put Myself in Your Shoes: Lifting the Egocentric Perspective from Exocentric Videoscited by 32
  11. 2024CyberDemo: Augmenting Simulated Human Demonstration for Real-World Dexterous Manipulationcited by 25
  12. 2024Cross-Embodiment Dexterous Grasping with Reinforcement Learningcited by 24
  13. 2025ViPRA: Video Prediction for Robot Actionscited by 24
  14. 2023GraspGF: Learning Score-based Grasping Primitive for Human-assisting Dexterous Graspingcited by 23
  15. 2023SparseDFF: Sparse-View Feature Distillation for One-Shot Dexterous Manipulationcited by 22
  16. 2022Contact2Grasp: 3D Grasp Synthesis via Hand-Object Contact Constraintcited by 21
  17. 2022DexTransfer: Real World Multi-fingered Dexterous Grasping with Minimal Human Demonstrationscited by 19
  18. 2022DualAfford: Learning Collaborative Visual Affordance for Dual-gripper Manipulationcited by 18
  19. 2025AMPLIFY: Actionless Motion Priors for Robot Learning from Videoscited by 18
  20. 2022Efficient Annotation and Learning for 3D Hand Pose Estimation: A Surveycited by 17
  21. 2024ExpertAF: Expert Actionable Feedback from Videocited by 17
  22. 2024ScrewMimic: Bimanual Imitation from Human Videos with Screw Space Projectioncited by 16
  23. 2024DexHandDiff: Interaction-aware Diffusion Planning for Adaptive Dexterous Manipulationcited by 15
  24. 2025DEXOP: A Device for Robotic Transfer of Dexterous Human Manipulationcited by 15
  25. 2025AnyDexGrasp: General Dexterous Grasping for Different Hands with Human-level Learning Efficiencycited by 14
  26. 2025Mitty: Diffusion-based Human-to-Robot Video Generationcited by 13
  27. 20233D Reconstruction of Objects in Hands without Real World 3D Supervisioncited by 12
  28. 2024Learning Diverse Bimanual Dexterous Manipulation Skills from Human Demonstrationscited by 12
  29. 2025MAPLE: Encoding Dexterous Robotic Manipulation Priors Learned From Egocentric Videoscited by 12
  30. 2025X-Humanoid: Robotize Human Videos to Generate Humanoid Videos at Scalecited by 12
  31. 2024One-Shot Imitation under Mismatched Executioncited by 11
  32. 2025HORT: Monocular Hand-held Objects Reconstruction with Transformerscited by 11
  33. 2023Learning a Universal Human Prior for Dexterous Manipulation from Human Preferencecited by 10
  34. 2024Learning by Watching: A Review of Video-based Learning Approaches for Robot Manipulationcited by 10
  35. 2024QuasiSim: Parameterized Quasi-Physical Simulators for Dexterous Manipulations Transfercited by 10
  36. 2023ForceSight: Text-Guided Mobile Manipulation with Visual-Force Goalscited by 8
  37. 2024Efficient Residual Learning with Mixture-of-Experts for Universal Dexterous Graspingcited by 8
  38. 2025H2R-Grounder: A Paired-Data-Free Paradigm for Translating Human Interaction Videos into Physically Grounded Robot Videoscited by 8

50 most cited of 83 All 83 (JSON)