mlsanityData to 7 Oct 2026

Holo-Dex: Teaching Dexterity with Immersive Mixed Reality

arXiv 2210.06463 ↗First posted 12 Oct 2022cs.RO cs.AI cs.CVcited by 52

Not on the map. Why?

Abstract (arXiv)

A fundamental challenge in teaching robots is to provide an effective interface for human teachers to demonstrate useful skills to a robot. This challenge is exacerbated in dexterous manipulation, where teaching high-dimensional, contact-rich behaviors often require esoteric teleoperation tools. In this work, we present Holo-Dex, a framework for dexterous manipulation that places a teacher in an immersive mixed reality through commodity VR headsets. The high-fidelity hand pose estimator onboard the headset is used to teleoperate the robot and collect demonstrations for a variety of general-purpose dexterous tasks. Given these demonstrations, we use powerful feature learning combined with non-parametric imitation to train dexterous skills. Our experiments on six common dexterous tasks, including in-hand rotation, spinning, and bottle opening, indicate that Holo-Dex can both collect high-quality demonstration data and train skills in a matter of hours. Finally, we find that our trained skills can exhibit generalization on objects not seen in training. Videos of Holo-Dex are available at https://holo-dex.github.io.

Key facts. Cited by 52 papers in the map (data through 7 Oct 2026). Builds on 52 of its 72 references that are in the map. First posted on arXiv 12 Oct 2022.

Builds on

52 of 72 references.

References · 52

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  4. 2015Continuous control with deep reinforcement learningcited by 3,279
  5. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  6. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  7. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  8. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  9. 2016End to End Learning for Self-Driving Carscited by 1,000
  10. 2019Solving Rubik's Cube with a Robot Handcited by 699
  11. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  12. 2018Learning Dexterous In-Hand Manipulationcited by 643
38 more
  1. 2021Implicit Behavioral Cloningcited by 442
  2. 2018Dynamic Few-Shot Visual Learning without Forgettingcited by 385
  3. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  4. 2016Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collectioncited by 312
  5. 2017Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperationcited by 286
  6. 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 241
  7. 2016CAD2RL: Real Single-Image Flight without a Single Real Imagecited by 227
  8. 2015Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hourscited by 208
  9. 2017Asymmetric Actor Critic for Image-Based Robot Learningcited by 202
  10. 2019Deep Dynamics Models for Learning Dexterous Manipulationcited by 200
  11. 2020MediaPipe Hands: On-device Real-time Hand Trackingcited by 157
  12. 2021A System for General In-Hand Object Re-Orientationcited by 155
  13. 2021The Surprising Effectiveness of Representation Learning for Visual Imitationcited by 149
  14. 2021Self-Supervised Representation Learning: Introduction, Advances and Challengescited by 147
  15. 2019End-to-End Robotic Reinforcement Learning without Reward Engineeringcited by 115
  16. 2020Visual Imitation Made Easycited by 111
  17. 2019DexPilot: Vision Based Teleoperation of Dexterous Robotic Hand-Arm Systemcited by 105
  18. 2017Long Text Generation via Adversarial Training with Leaked Informationcited by 100
  19. 2020State-Only Imitation Learning for Dexterous Manipulationcited by 90
  20. 2020RL-CycleGAN: Reinforcement Learning Aware Simulation-To-Realcited by 89
  21. 2022From One Hand to Multiple Hands: Imitation Learning for Dexterous Manipulation from Single-Camera Teleoperationcited by 79
  22. 2019Learning to Manipulate Deformable Objects without Demonstrationscited by 76
  23. 2022A Walk in the Park: Learning to Walk in 20 Minutes With Model-Free Reinforcement Learningcited by 76
  24. 2017Control of a Quadrotor with Reinforcement Learningcited by 75
  25. 2020An empirical investigation of the challenges of real-world reinforcement learningcited by 75
  26. 2022Dexterous Imitation Made Easy: A Learning-Based Framework for Efficient Dexterous Manipulationcited by 75
  27. 2022Watch and Match: Supercharging Imitation with Regularized Optimal Transportcited by 64
  28. 2021Generalization in Dexterous Manipulation via Geometry-Aware Multi-Task Learningcited by 49
  29. 2020RLOC: Terrain-Aware Legged Locomotion using Reinforcement Learning and Optimal Controlcited by 46
  30. 2015Learning Deep Control Policies for Autonomous Aerial Vehicles with MPC-Guided Policy Searchcited by 41
  31. 2022Iterative Residual Policy: for Goal-Conditioned Dynamic Manipulation of Deformable Objectscited by 36
  32. 2018A Practical Approach to Insertion with Variable Socket Position Using Deep Reinforcement Learningcited by 35
  33. 2017Learning to Fly by Crashingcited by 33
  34. 2016Learning Dexterous Manipulation Policies from Experience and Imitationcited by 32
  35. 2022Combining Learning-based Locomotion Policy with Model-based Manipulation for Legged Mobile Manipulatorscited by 32
  36. 2018Vision-based Teleoperation of Shadow Dexterous Hand using End-to-End Deep Neural Networkcited by 19
  37. 2020SARG: A Novel Semi Autoregressive Generator for Multi-turn Incomplete Utterance Restorationcited by 18
  38. 2020A Mobile Robot Hand-Arm Teleoperation System by Vision and IMUcited by 15

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 11; 2024: 22; 2025: 11; 2026: 7.

52 papers.

Sort:

By area:

  1. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  2. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  3. 2024HumanPlus: Humanoid Shadowing and Imitation from Humanscited by 164
  4. 2024EgoMimic: Scaling Imitation Learning via Egocentric Videocited by 160
  5. 2024DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulationcited by 149
  6. 2024Open-TeleVision: Teleoperation with Immersive Active Visual Feedbackcited by 137
  7. 2023AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation Systemcited by 136
  8. 2023On Bringing Robots Homecited by 101
  9. 2025DexGraspVLA: A Vision-Language-Action Framework Towards General Dexterous Graspingcited by 87
  10. 2023Rotating without Seeing: Towards In-hand Dexterity through Touchcited by 78
  11. 2023Teach a Robot to FISH: Versatile Imitation from One Minute of Demonstrationscited by 74
  12. 2024Learning Visuotactile Skills with Two Multifingered Handscited by 71
38 more
  1. 2024Bunny-VisionPro: Real-Time Bimanual Dexterous Teleoperation for Imitation Learningcited by 69
  2. 2023Dexterity from Touch: Self-Supervised Pre-Training of Tactile Representations with Robotic Playcited by 60
  3. 2023Robot Synesthesia: In-Hand Manipulation with Visuotactile Sensingcited by 51
  4. 2024Robot Utility Models: General Policies for Zero-Shot Deployment in New Environmentscited by 43
  5. 2024DynaMo: In-Domain Dynamics Pretraining for Visuo-Motor Controlcited by 43
  6. 2024Twisting Lids Off with Two Handscited by 36
  7. 2023See to Touch: Learning Tactile Dexterity through Visual Incentivescited by 33
  8. 2024ViViDex: Learning Vision-based Dexterous Manipulation from Human Videoscited by 33
  9. 2022Learning to Imitate Object Interactions from Internet Videoscited by 32
  10. 2024ACE: A Cross-Platform Visual-Exoskeletons System for Low-Cost Dexterous Teleoperationcited by 32
  11. 2023MimicTouch: Leveraging Multi-modal Human Tactile Demonstrations for Contact-rich Manipulationcited by 30
  12. 2024CyberDemo: Augmenting Simulated Human Demonstration for Real-World Dexterous Manipulationcited by 25
  13. 2024Bridging the Human to Robot Dexterity Gap through Object-Oriented Rewardscited by 25
  14. 2024Lessons from Learning to Spin "Pens"cited by 22
  15. 2024Bimanual Dexterity for Complex Taskscited by 22
  16. 2025Dexterous Manipulation through Imitation Learning: A Surveycited by 18
  17. 2024Hierarchical State Space Models for Continuous Sequence-to-Sequence Modelingcited by 15
  18. 2025Dexterity from Smart Lenses: Multi-Fingered Robot Manipulation with In-the-Wild Human Demonstrationscited by 14
  19. 2024Learning In-Hand Translation Using Tactile Skin With Shear and Normal Force Sensingcited by 13
  20. 2023REBOOT: Reuse Data for Bootstrapping Efficient Real-World Dexterous Manipulationcited by 12
  21. 2024Human-Agent Joint Learning for Efficient Robot Manipulation Skill Acquisitioncited by 10
  22. 2025Learning to Transfer Human Hand Skills for Robot Manipulationscited by 10
  23. 2026SimToolReal: An Object-Centric Policy for Zero-Shot Dexterous Tool Manipulationcited by 10
  24. 2023OpenVR: Teleoperation for Manipulationcited by 8
  25. 2025From Simple to Complex Skills: The Case of In-Hand Object Reorientationcited by 8
  26. 2025SR-Reward: Taking The Path More Traveledcited by 3
  27. 2025Interactive Imitation Learning for Dexterous Robotic Manipulation: Challenges and Perspectives -- A Surveycited by 3
  28. 2024Dexterous Manipulation Based on Prior Dexterous Grasp Pose Knowledgecited by 2
  29. 2025IRIS: An Immersive Robot Interaction Systemcited by 2
  30. 2026Human Universal Graspingcited by 2
  31. 2025TeleOpBench: A Simulator-Centric Benchmark for Dual-Arm Dexterous Teleoperationcited by 1
  32. 2025iFlyBot-VLA Technical Reportcited by 1
  33. 2026Mobile UMI: Cross-View Diffusion Policy with Decoupled Kinematics for Mobile Manipulationcited by 1
  34. 2026Play2Perfect: What Matters in Dexterous Play Pretraining for Precise Assembly?cited by 1
  35. 2024Self-supervised 6-DoF Robot Grasping by Demonstration via Augmented Reality Teleoperation Systemcited by 0
  36. 2024RAMPA: Robotic Augmented Reality for Machine Programming by DemonstrAtioncited by 0
  37. 2025Object-Focus Actor for Data-efficient Robot Generalization Dexterous Manipulationcited by 0
  38. 2026C2Dex: Contact-Consistent Reconstruction and Retargeting for Dexterous Manipulation from Monocular Videocited by 0

50 most cited of 52 All 52 (JSON)