mlsanityData to 7 Oct 2026

From One Hand to Multiple Hands: Imitation Learning for Dexterous Manipulation from Single-Camera Teleoperation

arXiv 2204.12490 ↗First posted 26 Apr 2022cs.RO cs.CV cs.LGcited by 79

On the map: Offline RL and imitation learning from data → Robot manipulation imitation learning

Abstract (arXiv)

We propose to perform imitation learning for dexterous manipulation with multi-finger robot hand from human demonstrations, and transfer the policy to the real robot hand. We introduce a novel single-camera teleoperation system to collect the 3D demonstrations efficiently with only an iPad and a computer. One key contribution of our system is that we construct a customized robot hand for each user in the physical simulator, which is a manipulator resembling the same kinematics structure and shape of the operator's hand. This provides an intuitive interface and avoid unstable human-robot hand retargeting for data collection, leading to large-scale and high quality data. Once the data is collected, the customized robot hand trajectories can be converted to different specified robot hands (models that are manufactured) to generate training demonstrations. With imitation learning using our data, we show large improvement over baselines with multiple complex manipulation tasks. Importantly, we show our learned policy is significantly more robust when transferring to the real robot. More videos can be found in the https://yzqin.github.io/dex-teleop-imitation .

Key facts. Cited by 79 papers in the map (data through 7 Oct 2026). Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 33 of its 57 references that are in the map. First posted on arXiv 26 Apr 2022.

Builds on

33 of 57 references.

Closest ancestors · 5

Same direction, newest first

  1. 2022Dexterous Imitation Made Easy: A Learning-Based Framework for Efficient Dexterous Manipulation24 Mar
  2. 2021DexMV: Imitation Learning for Dexterous Manipulation from Human Videos12 Aug
  3. 2020Visual Imitation Made Easy11 Aug
  4. 2020State-Only Imitation Learning for Dexterous Manipulation7 Apr
  5. 2018Zero-Shot Visual Imitation23 Apr

Other references · 28

  1. 2015Trust Region Policy Optimizationcited by 2,503
  2. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,600
  3. 2016Generative Adversarial Imitation Learningcited by 1,576
  4. 2019Solving Rubik's Cube with a Robot Handcited by 699
  5. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  6. 2018Learning Dexterous In-Hand Manipulationcited by 643
  7. 2020SAPIEN: A SimulAted Part-based Interactive ENvironmentcited by 584
  8. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  9. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 361
  10. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
  11. 2017Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperationcited by 286
  12. 2018Generative Adversarial Imitation from Observationcited by 183
16 more
  1. 2021FrankMocap: A Monocular 3D Whole-Body Pose Estimation System via Regression and Integrationcited by 168
  2. 2017Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translationcited by 167
  3. 2020MediaPipe Hands: On-device Real-time Hand Trackingcited by 157
  4. 2019DexPilot: Vision Based Teleoperation of Dexterous Robotic Hand-Arm Systemcited by 105
  5. 2019Grasping in the Wild:Learning 6DoF Closed-Loop Grasping from Low-Cost Demonstrationscited by 104
  6. 2015Benchmarking in Manipulation Research: The YCB Object and Model Set and Benchmarking Protocolscited by 98
  7. 2018Playing hard exploration games by watching YouTubecited by 98
  8. 2021Learning by Watching: Physical Imitation of Manipulation Skills from Human Videoscited by 87
  9. 2020Reinforcement Learning with Videos: Combining Offline Observations with Interactioncited by 80
  10. 2022Robotic Telekinesis: Learning a Robotic Hand Imitator by Watching Humans on Youtubecited by 79
  11. 2018Multiple Interactions Made Easy (MIME): Large Scale Demonstrations Data for Imitationcited by 76
  12. 2020FrankMocap: Fast Monocular 3D Hand and Body Motion Capture by Regression and Integrationcited by 72
  13. 2019Graph-Structured Visual Imitationcited by 43
  14. 2020Physics-Based Dexterous Manipulations with Estimated Hand Poses and Residual Reinforcement Learningcited by 37
  15. 2018Task-Oriented Hand Motion Retargeting for Dexterous Manipulation Imitationcited by 25
  16. 2018Vision-based Teleoperation of Shadow Dexterous Hand using End-to-End Deep Neural Networkcited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 10; 2023: 18; 2024: 27; 2025: 17; 2026: 7.

79 papers.

Sort:

By area:

  1. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  2. 2024DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulationcited by 149
  3. 2024Open-TeleVision: Teleoperation with Immersive Active Visual Feedbackcited by 137
  4. 2023AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation Systemcited by 136
  5. 2022DexGraspNet: A Large-Scale Robotic Dexterous Grasp Dataset for General Objects Based on Simulationcited by 129
  6. 2022Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learningcited by 113
  7. 2022VideoDex: Learning Dexterity from Internet Videoscited by 101
  8. 2023General In-Hand Object Rotation with Vision and Touchcited by 87
  9. 2025DexGraspVLA: A Vision-Language-Action Framework Towards General Dexterous Graspingcited by 87
  10. 2024TACO: Benchmarking Generalizable Bimanual Tool-ACtion-Object Understandingcited by 85
  11. 2022DexPoint: Generalizable Point Cloud Reinforcement Learning for Sim-to-Real Dexterous Manipulationcited by 78
  12. 2023Rotating without Seeing: Towards In-hand Dexterity through Touchcited by 78
38 more
  1. 2023LEAP Hand: Low-Cost, Efficient, and Anthropomorphic Hand for Robot Learningcited by 78
  2. 2024OAKINK2: A Dataset of Bimanual Hands-Object Manipulation in Complex Task Completioncited by 78
  3. 2024ExBody2: Advanced Expressive Humanoid Whole-Body Controlcited by 75
  4. 2024Learning Visuotactile Skills with Two Multifingered Handscited by 71
  5. 2024Bunny-VisionPro: Real-Time Bimanual Dexterous Teleoperation for Imitation Learningcited by 69
  6. 2023DexArt: Benchmarking Generalizable Dexterous Manipulation with Articulated Objectscited by 64
  7. 2024Policy Decorator: Model-Agnostic Online Refinement for Large Policy Modelcited by 60
  8. 2025ManipTrans: Efficient Dexterous Bimanual Manipulation Transfer via Residual Learningcited by 56
  9. 2022Holo-Dex: Teaching Dexterity with Immersive Mixed Realitycited by 52
  10. 2023Robot Synesthesia: In-Hand Manipulation with Visuotactile Sensingcited by 51
  11. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  12. 2024Deep Learning-Based Object Pose Estimation: A Comprehensive Surveycited by 39
  13. 2024Hand-Object Interaction Pretraining from Videoscited by 38
  14. 2024Twisting Lids Off with Two Handscited by 36
  15. 2022Learning Continuous Grasping Function with a Dexterous Hand from Human Demonstrationscited by 34
  16. 2023Dynamic Handover: Throw and Catch with Bimanual Handscited by 33
  17. 2022Learning to Imitate Object Interactions from Internet Videoscited by 32
  18. 2024ACE: A Cross-Platform Visual-Exoskeletons System for Low-Cost Dexterous Teleoperationcited by 32
  19. 2025BEHAVIOR Robot Suite: Streamlining Real-World Whole-Body Manipulation for Everyday Household Activitiescited by 29
  20. 2024DITTO: Demonstration Imitation by Trajectory Transformationcited by 28
  21. 2024CyberDemo: Augmenting Simulated Human Demonstration for Real-World Dexterous Manipulationcited by 25
  22. 2025SPIDER: Scalable Physics-Informed Dexterous Retargetingcited by 25
  23. 2024Cross-Embodiment Dexterous Grasping with Reinforcement Learningcited by 24
  24. 2024Ag2Manip: Learning Novel Manipulation Skills with Agent-Agnostic Visual and Action Representationscited by 23
  25. 2024Lessons from Learning to Spin "Pens"cited by 22
  26. 2023Chain-of-Thought Predictive Controlcited by 18
  27. 2025Dexterous Manipulation through Imitation Learning: A Surveycited by 18
  28. 2023ContactArt: Learning 3D Interaction Priors for Category-level Articulated Object and Hand Poses Estimationcited by 16
  29. 2023Novel-view Synthesis and Pose Estimation for Hand-Object Interaction from Sparse Viewscited by 16
  30. 2025DEXOP: A Device for Robotic Transfer of Dexterous Human Manipulationcited by 15
  31. 2023ViHOPE: Visuotactile In-Hand Object 6D Pose Estimation with Shape Completioncited by 14
  32. 2024Learning In-Hand Translation Using Tactile Skin With Shear and Normal Force Sensingcited by 13
  33. 20233D Reconstruction of Objects in Hands without Real World 3D Supervisioncited by 12
  34. 2024Learning Diverse Bimanual Dexterous Manipulation Skills from Human Demonstrationscited by 12
  35. 2024QuasiSim: Parameterized Quasi-Physical Simulators for Dexterous Manipulations Transfercited by 10
  36. 2024Guided Reinforcement Learning for Robust Multi-Contact Loco-Manipulationcited by 9
  37. 2025Dexplore: Scalable Neural Control for Dexterous Manipulation from Reference-Scoped Explorationcited by 9
  38. 2026Do as I Do: Dexterous Manipulation Data from Everyday Human Videoscited by 9

50 most cited of 79 All 79 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Tilde: Teleoperation for Dexterous In-Hand Manipulation Learning with a DeltaHand
  2. 2025Dexterous Teleoperation of 20-DoF ByteDexter Hand via Human Motion Retargeting
  3. 2025Open TeleDex: A Hardware-Agnostic Teleoperation System for Imitation Learning based Dexterous Manipulation
  4. 2022Evaluating the Effectiveness of Corrective Demonstrations and a Low-Cost Sensor for Dexterous Manipulation
  5. 2026Dexterous Point Policy: Learning Point-based Dexterous Hand Policies from Human Demonstrations
  6. 2020Learning Multi-Arm Manipulation Through Collaborative Teleoperation
  7. 2022Learning Generalizable Dexterous Manipulation from Human Grasp Affordance
  8. 2025RwoR: Generating Robot Demonstrations from Human Hand Collection for Policy Learning without Robot