mlsanityData to 7 Oct 2026

DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulation

arXiv 2403.07788 ↗First posted 12 Mar 2024cs.RO cs.AI cs.CVcited by 149

On the map: Offline RL and imitation learning from data → Robot manipulation imitation learning

Abstract (arXiv)

Imitation learning from human hand motion data presents a promising avenue for imbuing robots with human-like dexterity in real-world manipulation tasks. Despite this potential, substantial challenges persist, particularly with the portability of existing hand motion capture (mocap) systems and the complexity of translating mocap data into effective robotic policies. To tackle these issues, we introduce DexCap, a portable hand motion capture system, alongside DexIL, a novel imitation algorithm for training dexterous robot skills directly from human hand mocap data. DexCap offers precise, occlusion-resistant tracking of wrist and finger motions based on SLAM and electromagnetic field together with 3D observations of the environment. Utilizing this rich dataset, DexIL employs inverse kinematics and point cloud-based imitation learning to seamlessly replicate human actions with robot hands. Beyond direct learning from human motion, DexCap also offers an optional human-in-the-loop correction mechanism during policy rollouts to refine and further improve task performance. Through extensive evaluation across six challenging dexterous manipulation tasks, our approach not only demonstrates superior performance but also showcases the system's capability to effectively learn from in-the-wild mocap data, paving the way for future data collection methods in the pursuit of human-level robot dexterity. More details can be found at https://dex-cap.github.io

Key facts. Cited by 149 papers in the map (data through 7 Oct 2026). Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 76 of its 115 references that are in the map. First posted on arXiv 12 Mar 2024.

Builds on

76 of 115 references.

Closest ancestors · 13

Same direction, newest first

  1. 20243D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representations6 Mar
  2. 2023GELLO: A General, Low-Cost, and Intuitive Teleoperation Framework for Robot Manipulators22 Sep
  3. 2023XSkill: Cross Embodiment Skill Discovery19 Jul
  4. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware23 Apr
  5. 2023MimicPlay: Long-Horizon Imitation Learning by Watching Human Play24 Feb
  6. 2022VIOLA: Imitation Learning for Vision-Based Manipulation with Object Proposal Priors20 Oct
  7. 2022From One Hand to Multiple Hands: Imitation Learning for Dexterous Manipulation from Single-Camera Teleoperation26 Apr
  8. 2022Dexterous Imitation Made Easy: A Learning-Based Framework for Efficient Dexterous Manipulation24 Mar
  9. 2021DexMV: Imitation Learning for Dexterous Manipulation from Human Videos12 Aug
  10. 2021Generalization Through Hand-Eye Coordination: An Action Space for Learning Spatially-Invariant Visuomotor Control28 Feb
  11. 2020Human-in-the-Loop Imitation Learning using Remote Teleoperation12 Dec
  12. 2020Visual Imitation Made Easy11 Aug
1 more
  1. 2019Self-Supervised Correspondence in Visuomotor Policy Learning16 Sep

Other references · 63

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2020Denoising Diffusion Implicit Modelscited by 9,655
  4. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  5. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  6. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  7. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  8. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  9. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  10. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  11. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  12. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
38 more
  1. 2020GRAB: A Dataset of Whole-Body Human Grasping of Objectscited by 397
  2. 2024Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robotscited by 373
  3. 2021DexYCB: A Benchmark for Capturing Hand Grasping of Objectscited by 320
  4. 2022ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulationcited by 306
  5. 2023Reconstructing Hands in 3D with Transformerscited by 298
  6. 2017Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperationcited by 286
  7. 2019FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Imagescited by 271
  8. 2022Masked Visual Pre-training for Motor Controlcited by 246
  9. 2020InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Imagecited by 244
  10. 2017One-Shot Visual Imitation Learning via Meta-Learningcited by 225
  11. 2022Human-to-Robot Imitation in the Wildcited by 157
  12. 2021A System for General In-Hand Object Re-Orientationcited by 155
  13. 2022Physical Inertial Poser (PIP): Physics-aware Real-time Human Motion Tracking from Sparse Inertial Sensorscited by 144
  14. 2022DeXtreme: Transfer of Agile In-hand Manipulation from Simulation to Realitycited by 136
  15. 2023RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketchescited by 131
  16. 2019DexPilot: Vision Based Teleoperation of Dexterous Robotic Hand-Arm Systemcited by 105
  17. 2019AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videoscited by 105
  18. 2019Grasping in the Wild:Learning 6DoF Closed-Loop Grasping from Low-Cost Demonstrationscited by 104
  19. 2022In-Hand Object Rotation via Rapid Motor Adaptationcited by 102
  20. 2022Robot Learning on the Job: Human-in-the-Loop Autonomy and Learning During Deploymentcited by 101
  21. 2022VideoDex: Learning Dexterity from Internet Videoscited by 101
  22. 2023On Bringing Robots Homecited by 101
  23. 2021Reset-Free Reinforcement Learning via Multi-Task Learning: Learning Dexterous Manipulation Behaviors without Human Interventioncited by 90
  24. 2022DexVIP: Learning Dexterous Grasping with Human Hand Pose Priors from Videocited by 83
  25. 2022Robotic Telekinesis: Learning a Robotic Hand Imitator by Watching Humans on Youtubecited by 79
  26. 2022DexPoint: Generalizable Point Cloud Reinforcement Learning for Sim-to-Real Dexterous Manipulationcited by 78
  27. 2023Rotating without Seeing: Towards In-hand Dexterity through Touchcited by 78
  28. 2023LEAP Hand: Low-Cost, Efficient, and Anthropomorphic Hand for Robot Learningcited by 78
  29. 2022Transformer Inertial Poser: Real-time Human Motion Reconstruction from Sparse IMUs with Simultaneous Terrain Generationcited by 75
  30. 2023Dexterity from Touch: Self-Supervised Pre-Training of Tactile Representations with Robotic Playcited by 60
  31. 2022Holo-Dex: Teaching Dexterity with Immersive Mixed Realitycited by 52
  32. 2023Towards Generalizable Zero-Shot Manipulation via Translating Human Interaction Planscited by 51
  33. 2023AirExo: Low-Cost Exoskeletons for Learning Whole-Arm Manipulation in the Wildcited by 46
  34. 2023Sequential Dexterity: Chaining Dexterous Policies for Long-Horizon Manipulationcited by 43
  35. 2017A survey of robot learning from demonstrations for Human-Robot Collaborationcited by 38
  36. 2023Stabilize to Act: Learning to Coordinate for Bimanual Manipulationcited by 36
  37. 2024Twisting Lids Off with Two Handscited by 36
  38. 2024Efficient Data Collection for Robotic Manipulation via Compositional Generalizationcited by 35

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 34; 2025: 64; 2026: 51.

149 papers.

Sort:

By area:

  1. 2025EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Videocited by 191
  2. 2024RoboMIND: Benchmark on Multi-embodiment Intelligence Normative Data for Robot Manipulationcited by 166
  3. 2024HumanPlus: Humanoid Shadowing and Imitation from Humanscited by 164
  4. 2024EgoMimic: Scaling Imitation Learning via Egocentric Videocited by 160
  5. 2024Open-TeleVision: Teleoperation with Immersive Active Visual Feedbackcited by 137
  6. 2024Flow as the Cross-Domain Manipulation Interfacecited by 126
  7. 2026DreamDojo: A Generalist Robot World Model from Large-Scale Human Videoscited by 117
  8. 2025DexGraspVLA: A Vision-Language-Action Framework Towards General Dexterous Graspingcited by 87
  9. 2024Manipulate-Anything: Automating Real-World Robots using Vision-Language Modelscited by 82
  10. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  11. 2024ExBody2: Advanced Expressive Humanoid Whole-Body Controlcited by 75
  12. 2024Learning Visuotactile Skills with Two Multifingered Handscited by 71
38 more
  1. 2024Bunny-VisionPro: Real-Time Bimanual Dexterous Teleoperation for Imitation Learningcited by 69
  2. 20243D-ViTac: Learning Fine-Grained Manipulation with Visuo-Tactile Sensingcited by 67
  3. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Surveycited by 66
  4. 2025HOMIE: Humanoid Loco-Manipulation with Isomorphic Exoskeleton Cockpitcited by 65
  5. 2025Humanoid Policy ~ Human Policycited by 65
  6. 2025Scalable Vision-Language-Action Model Pretraining for Robotic Manipulation with Real-Life Human Activity Videoscited by 63
  7. 2024OKAMI: Teaching Humanoid Robots Manipulation Skills through Single Video Imitationcited by 62
  8. 2025ManipTrans: Efficient Dexterous Bimanual Manipulation Transfer via Residual Learningcited by 56
  9. 2024TRANSIC: Sim-to-Real Policy Transfer by Learning from Online Correctioncited by 50
  10. 2024Generalizable Humanoid Manipulation with 3D Diffusion Policiescited by 49
  11. 2025VidBot: Learning Generalizable 3D Actions from In-the-Wild 2D Human Videos for Zero-Shot Robotic Manipulationcited by 46
  12. 2025NavDP: Learning Sim-to-Real Navigation Diffusion Policy with Privileged Information Guidancecited by 46
  13. 2026World Action Models: The Next Frontier in Embodied AIcited by 46
  14. 2025H-RDT: Human Manipulation Enhanced Bimanual Robotic Manipulationcited by 41
  15. 2025Diffusion Models for Robotic Manipulation: A Surveycited by 39
  16. 2024Hand-Object Interaction Pretraining from Videoscited by 38
  17. 2024R+X: Retrieval and Execution from Everyday Human Videoscited by 36
  18. 2024ForceMimic: Force-Centric Imitation Learning with Force-Motion Capture System for Contact-Rich Manipulationcited by 36
  19. 2025Emergence of Human to Robot Transfer in Vision-Language-Action Modelscited by 36
  20. 2025DexWild: Dexterous Human Interactions for In-the-Wild Robot Policiescited by 34
  21. 2024ViViDex: Learning Vision-based Dexterous Manipulation from Human Videoscited by 33
  22. 2024ACE: A Cross-Platform Visual-Exoskeletons System for Low-Cost Dexterous Teleoperationcited by 32
  23. 2024ARCap: Collecting High-quality Human Demonstrations for Robot Learning with Augmented Reality Feedbackcited by 31
  24. 2025DexMachina: Functional Retargeting for Bimanual Dexterous Manipulationcited by 30
  25. 2025BEHAVIOR Robot Suite: Streamlining Real-World Whole-Body Manipulation for Everyday Household Activitiescited by 29
  26. 2024SPA: 3D Spatial-Awareness Enables Effective Embodied Representationcited by 28
  27. 2025EgoZero: Robot Learning from Smart Glassescited by 28
  28. 2025Crossing the Human-Robot Embodiment Gap with Sim-to-Real RL using One Human Demonstrationcited by 27
  29. 2024Bridging the Human to Robot Dexterity Gap through Object-Oriented Rewardscited by 25
  30. 2024Lessons from Learning to Spin "Pens"cited by 22
  31. 2024Bimanual Dexterity for Complex Taskscited by 22
  32. 2024VoxAct-B: Voxel-Based Acting and Stabilizing Policy for Bimanual Manipulationcited by 21
  33. 2025In-N-On: Scaling Egocentric Manipulation with in-the-wild and on-task Datacited by 21
  34. 2025Predicting 4D Hand Trajectory from Monocular Videoscited by 19
  35. 2025DexVLG: Dexterous Vision-Language-Grasp Model at Scalecited by 19
  36. 2025Dexterous Manipulation through Imitation Learning: A Surveycited by 18
  37. 2025AirExo-2: Scaling up Generalizable Robotic Imitation Learning with Low-Cost Exoskeletonscited by 17
  38. 2025ActiveUMI: Robotic Manipulation with Active Perception from Robot-Free Human Demonstrationscited by 17

50 most cited of 149 All 149 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Dexterous Point Policy: Learning Point-based Dexterous Hand Policies from Human Demonstrations
  2. 2025DexFlyWheel: A Scalable and Self-improving Data Generation Framework for Dexterous Manipulation
  3. 2025DexForce: Extracting Force-informed Actions from Kinesthetic Demonstrations for Dexterous Manipulation
  4. 2026From Grasps to Dexterity: Large-Scale Grasp Pretraining for Dexterous Manipulation
  5. 2025DexH2R: A Benchmark for Dynamic Dexterous Grasping in Human-to-Robot Handover
  6. 2025World Models for Learning Dexterous Hand-Object Interactions from Human Videos
  7. 2024DexMimicGen: Automated Data Generation for Bimanual Dexterous Manipulation via Imitation Learning
  8. 2026UniDex: A Robot Foundation Suite for Universal Dexterous Hand Control from Egocentric Human Videos