mlsanityData to 7 Oct 2026

DexMV: Imitation Learning for Dexterous Manipulation from Human Videos

arXiv 2108.05877 ↗First posted 12 Aug 2021cs.LG cs.CV cs.ROcited by 194

On the map: Offline RL and imitation learning from data → Robot manipulation imitation learning

Abstract (arXiv)

While significant progress has been made on understanding hand-object interactions in computer vision, it is still very challenging for robots to perform complex dexterous manipulation. In this paper, we propose a new platform and pipeline DexMV (Dexterous Manipulation from Videos) for imitation learning. We design a platform with: (i) a simulation system for complex dexterous manipulation tasks with a multi-finger robot hand and (ii) a computer vision system to record large-scale demonstrations of a human hand conducting the same tasks. In our novel pipeline, we extract 3D hand and object poses from videos, and propose a novel demonstration translation method to convert human motion to robot demonstrations. We then apply and benchmark multiple imitation learning algorithms with the demonstrations. We show that the demonstrations can indeed improve robot learning by a large margin and solve the complex tasks which reinforcement learning alone cannot solve. More details can be found in the project page: https://yzqin.github.io/dexmv

Key facts. Cited by 194 papers in the map (data through 7 Oct 2026). Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 62 of its 98 references that are in the map. First posted on arXiv 12 Aug 2021.

Builds on

62 of 98 references.

Closest ancestors · 4

Same direction, newest first

  1. 2020Visual Imitation Made Easy11 Aug
  2. 2020State-Only Imitation Learning for Dexterous Manipulation7 Apr
  3. 2019Third-Person Visual Imitation Learning via Decoupled Hierarchical Controller21 Nov
  4. 2018Zero-Shot Visual Imitation23 Apr

Other references · 58

  1. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  2. 2015Trust Region Policy Optimizationcited by 2,503
  3. 2016Generative Adversarial Imitation Learningcited by 1,576
  4. 2016End to End Learning for Self-Driving Carscited by 1,000
  5. 2017PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenescited by 768
  6. 2019Solving Rubik's Cube with a Robot Handcited by 699
  7. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  8. 2018Learning Dexterous In-Hand Manipulationcited by 643
  9. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  10. 2018Behavioral Cloning from Observationcited by 482
  11. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 453
  12. 2017Neural 3D Mesh Renderercited by 437
38 more
  1. 2020GRAB: A Dataset of Whole-Body Human Grasping of Objectscited by 397
  2. 2018PVNet: Pixel-wise Voting Network for 6DoF Pose Estimationcited by 385
  3. 2019Learning joint reconstruction of hands and manipulated objectscited by 377
  4. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 361
  5. 2021DexYCB: A Benchmark for Capturing Hand Grasping of Objectscited by 320
  6. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
  7. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  8. 2019FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Imagescited by 271
  9. 2017SSD-6D: Making RGB-based 3D detection and 6D pose estimation great againcited by 243
  10. 2017Learning to Estimate 3D Hand Pose from Single RGB Imagescited by 234
  11. 2017First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotationscited by 229
  12. 2020Learning Agile Robotic Locomotion Skills by Imitating Animalscited by 229
  13. 2017BB8: A Scalable, Accurate, Robust to Partial Occlusion Method for Predicting the 3D Poses of Challenging Objects without Using Depthcited by 225
  14. 2017Real-Time Seamless Single Shot 6D Object Pose Predictioncited by 210
  15. 2019PVN3D: A Deep Point-wise 3D Keypoints Voting Network for 6DoF Pose Estimationcited by 204
  16. 20193D Hand Shape and Pose from Images in the Wildcited by 190
  17. 2018Generative Adversarial Imitation from Observationcited by 183
  18. 2020ContactPose: A Dataset of Grasps with Object Contact and Hand Posecited by 180
  19. 20193D Hand Shape and Pose Estimation from a Single RGB Imagecited by 174
  20. 2017Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translationcited by 167
  21. 2021Hand-Object Contact Consistency Reasoning for Human Grasps Generationcited by 160
  22. 2020Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wildcited by 146
  23. 2019ContactDB: Analyzing and Predicting Grasp Contact via Thermal Imagingcited by 145
  24. 2021Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Timecited by 140
  25. 2019Pushing the Envelope for RGB-based Dense 3D Hand Pose Estimation via Neural Renderingcited by 115
  26. 2018Hand Pose Estimation via Latent 2.5D Heatmap Regressioncited by 111
  27. 2019DexPilot: Vision Based Teleoperation of Dexterous Robotic Hand-Arm Systemcited by 105
  28. 2019AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videoscited by 105
  29. 2018Segmentation-driven 6D Object Pose Estimationcited by 104
  30. 2019Grasping in the Wild:Learning 6DoF Closed-Loop Grasping from Low-Cost Demonstrationscited by 104
  31. 2015Benchmarking in Manipulation Research: The YCB Object and Model Set and Benchmarking Protocolscited by 98
  32. 2018Playing hard exploration games by watching YouTubecited by 98
  33. 2018Cross-modal Deep Variational Hand Pose Estimationcited by 89
  34. 2019ContactGrasp: Functional Multi-finger Grasp Synthesis from Contactcited by 87
  35. 2021Learning by Watching: Physical Imitation of Manipulation Skills from Human Videoscited by 87
  36. 2020Reinforcement Learning with Videos: Combining Offline Observations with Interactioncited by 80
  37. 2018Multiple Interactions Made Easy (MIME): Large Scale Demonstrations Data for Imitationcited by 76
  38. 2019State Alignment-based Imitation Learningcited by 73

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 20; 2023: 26; 2024: 49; 2025: 56; 2026: 42.

194 papers.

Sort:

By area:

  1. 2022HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interactioncited by 321
  2. 2024Latent Action Pretraining from Videoscited by 302
  3. 2022Masked Visual Pre-training for Motor Controlcited by 246
  4. 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 221
  5. 2025DexVLA: Vision-Language Model with Plug-In Diffusion Expert for General Robot Controlcited by 191
  6. 2025DreamGen: Unlocking Generalization in Robot Learning through Video World Modelscited by 168
  7. 2024HumanPlus: Humanoid Shadowing and Imitation from Humanscited by 164
  8. 2024DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulationcited by 149
  9. 2023Structured World Models from Human Videoscited by 144
  10. 2024Data Scaling Laws in Imitation Learning for Robotic Manipulationcited by 144
  11. 2024Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulationcited by 142
  12. 2023AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation Systemcited by 136
38 more
  1. 2022OakInk: A Large-scale Knowledge Repository for Understanding Hand-Object Interactioncited by 129
  2. 2024Flow as the Cross-Domain Manipulation Interfacecited by 126
  3. 2023UniDexGrasp: Universal Robotic Dexterous Grasping via Learning Diverse Proposal Generation and Goal-Conditioned Policycited by 117
  4. 2024Track2Act: Predicting Point Tracks from Internet Videos enables Generalizable Robot Manipulationcited by 116
  5. 20243D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representationscited by 115
  6. 2022Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learningcited by 113
  7. 2023Affordance Diffusion: Synthesizing Hand-Object Interactionscited by 110
  8. 2021D-Grasp: Physically Plausible Dynamic Grasp Synthesis for Hand-Object Interactionscited by 109
  9. 2025FLARE: Robot Learning with Implicit World Modelingcited by 105
  10. 2022VideoDex: Learning Dexterity from Internet Videoscited by 101
  11. 2023General In-Hand Object Rotation with Vision and Touchcited by 87
  12. 2025DexGraspVLA: A Vision-Language-Action Framework Towards General Dexterous Graspingcited by 87
  13. 2024TACO: Benchmarking Generalizable Bimanual Tool-ACtion-Object Understandingcited by 85
  14. 2022From One Hand to Multiple Hands: Imitation Learning for Dexterous Manipulation from Single-Camera Teleoperationcited by 79
  15. 2022DexPoint: Generalizable Point Cloud Reinforcement Learning for Sim-to-Real Dexterous Manipulationcited by 78
  16. 2023Rotating without Seeing: Towards In-hand Dexterity through Touchcited by 78
  17. 2023LEAP Hand: Low-Cost, Efficient, and Anthropomorphic Hand for Robot Learningcited by 78
  18. 2024OAKINK2: A Dataset of Bimanual Hands-Object Manipulation in Complex Task Completioncited by 78
  19. 2022Dexterous Imitation Made Easy: A Learning-Based Framework for Efficient Dexterous Manipulationcited by 75
  20. 2024Bunny-VisionPro: Real-Time Bimanual Dexterous Teleoperation for Imitation Learningcited by 69
  21. 2023DexArt: Benchmarking Generalizable Dexterous Manipulation with Articulated Objectscited by 64
  22. 2025Scalable Vision-Language-Action Model Pretraining for Robotic Manipulation with Real-Life Human Activity Videoscited by 63
  23. 2024DiffH2O: Diffusion-Based Synthesis of Hand-Object Interactions from Textual Descriptionscited by 60
  24. 2024GraspXL: Generating Grasping Motions for Diverse Objects at Scalecited by 59
  25. 2024Koala: Key frame-conditioned long video-LLMcited by 57
  26. 2025ManipTrans: Efficient Dexterous Bimanual Manipulation Transfer via Residual Learningcited by 56
  27. 2023ArtiGrasp: Physically Plausible Synthesis of Bi-Manual Dexterous Grasping and Articulationcited by 53
  28. 2023Physically Plausible Full-Body Hand-Object Interaction Synthesiscited by 52
  29. 2023Locomotion-Action-Manipulation: Synthesizing Human-Scene Interactions in Complex 3D Environmentscited by 51
  30. 2023Towards Generalizable Zero-Shot Manipulation via Translating Human Interaction Planscited by 51
  31. 2023Robot Synesthesia: In-Hand Manipulation with Visuotactile Sensingcited by 51
  32. 2023Zero-Shot Robot Manipulation from Passive Human Videoscited by 49
  33. 2022Learning Generalizable Dexterous Manipulation from Human Grasp Affordancecited by 46
  34. 2025VidBot: Learning Generalizable 3D Actions from In-the-Wild 2D Human Videos for Zero-Shot Robotic Manipulationcited by 46
  35. 2022Graph Inverse Reinforcement Learning from Diverse Videoscited by 45
  36. 2024DynaMo: In-Domain Dynamics Pretraining for Visuo-Motor Controlcited by 43
  37. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  38. 2024Hand-Object Interaction Pretraining from Videoscited by 38

50 most cited of 194 All 194 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Dexterous Point Policy: Learning Point-based Dexterous Hand Policies from Human Demonstrations
  2. 2026Human-to-Robot Interaction: Learning from Video Demonstration for Robot Imitation
  3. 2026AdvDex: Learning Dexterous Manipulation from Human Demonstrations via Joint-Aligned Actions and Adversarial Learning
  4. 2022Evaluating the Effectiveness of Corrective Demonstrations and a Low-Cost Sensor for Dexterous Manipulation
  5. 2025Robotic Manipulation via Imitation Learning: Taxonomy, Evolution, Benchmark, and Challenges
  6. 2021Coarse-to-Fine Imitation Learning: Robot Manipulation from a Single Demonstration
  7. 2026DexSIM: Real-time Dexterous Simulation with Unified Causal Video Diffusion
  8. 2025DexForce: Extracting Force-informed Actions from Kinesthetic Demonstrations for Dexterous Manipulation