mlsanityData to 13 Oct 2026

Robotic Telekinesis: Learning a Robotic Hand Imitator by Watching Humans on Youtube

arXiv 2202.10448 ↗First posted 21 Feb 2022cs.RO cs.AI cs.CVcited by 79

Not on the map. Why?

Abstract (arXiv)

We build a system that enables any human to control a robot hand and arm, simply by demonstrating motions with their own hand. The robot observes the human operator via a single RGB camera and imitates their actions in real-time. Human hands and robot hands differ in shape, size, and joint structure, and performing this translation from a single uncalibrated camera is a highly underconstrained problem. Moreover, the retargeted trajectories must effectively execute tasks on a physical robot, which requires them to be temporally smooth and free of self-collisions. Our key insight is that while paired human-robot correspondence data is expensive to collect, the internet contains a massive corpus of rich and diverse human hand videos. We leverage this data to train a system that understands human hands and retargets a human video stream into a robot hand-arm trajectory that is smooth, swift, safe, and semantically similar to the guiding demonstration. We demonstrate that it enables previously untrained people to teleoperate a robot on various dexterous manipulation tasks. Our low-cost, glove-free, marker-free remote teleoperation system makes robot teaching more accessible and we hope that it can aid robots in learning to act autonomously in the real world. Videos at https://robotic-telekinesis.github.io/

Key facts. Cited by 79 papers in the map (data through 13 Oct 2026). Builds on 22 of its 40 references that are in the map. First posted on arXiv 21 Feb 2022.

Builds on

22 of 40 references.

References · 22

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,950
  2. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,603
  3. 2018OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,161
  4. 2017End-to-end Recovery of Human Shape and Posecited by 883
  5. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 792
  6. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 687
  7. 2020GRAB: A Dataset of Whole-Body Human Grasping of Objectscited by 398
  8. 2019Learning joint reconstruction of hands and manipulated objectscited by 377
  9. 2021DexYCB: A Benchmark for Capturing Hand Grasping of Objectscited by 322
  10. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  11. 2019FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Imagescited by 271
  12. 2020Learning Agile Robotic Locomotion Skills by Imitating Animalscited by 229
10 more
  1. 2021FrankMocap: A Monocular 3D Whole-Body Pose Estimation System via Regression and Integrationcited by 168
  2. 2019ContactDB: Analyzing and Predicting Grasp Contact via Thermal Imagingcited by 145
  3. 2020Reconstructing Hand-Object Interactions in the Wildcited by 143
  4. 2021Collaborative Regression of Expressive Bodies using Moderationcited by 137
  5. 2019DexPilot: Vision Based Teleoperation of Dexterous Robotic Hand-Arm Systemcited by 106
  6. 2021XIRL: Cross-embodiment Inverse Reinforcement Learningcited by 102
  7. 2020State-Only Imitation Learning for Dexterous Manipulationcited by 90
  8. 2018Neural Kinematic Networks for Unsupervised Motion Retargettingcited by 86
  9. 2018Task-Oriented Hand Motion Retargeting for Dexterous Manipulation Imitationcited by 25
  10. 2018Vision-based Teleoperation of Shadow Dexterous Hand using End-to-End Deep Neural Networkcited by 19

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2022: 8; 2023: 15; 2024: 24; 2025: 19; 2026: 13.

79 papers.

Sort:

By area:

  1. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,512
  2. 2022DayDreamer: World Models for Physical Robot Learningcited by 375
  3. 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 222
  4. 2024HumanPlus: Humanoid Shadowing and Imitation from Humanscited by 166
  5. 2024EgoMimic: Scaling Imitation Learning via Egocentric Videocited by 165
  6. 2023Learning to Act from Actionless Videos through Dense Correspondencescited by 160
  7. 2024DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulationcited by 149
  8. 2024Open-TeleVision: Teleoperation with Immersive Active Visual Feedbackcited by 138
  9. 2023AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation Systemcited by 137
  10. 2022VideoDex: Learning Dexterity from Internet Videoscited by 101
  11. 2025DexGraspVLA: A Vision-Language-Action Framework Towards General Dexterous Graspingcited by 87
  12. 2022From One Hand to Multiple Hands: Imitation Learning for Dexterous Manipulation from Single-Camera Teleoperationcited by 79
38 more
  1. 2023LEAP Hand: Low-Cost, Efficient, and Anthropomorphic Hand for Robot Learningcited by 78
  2. 2022Dexterous Imitation Made Easy: A Learning-Based Framework for Efficient Dexterous Manipulationcited by 75
  3. 2024Learning Visuotactile Skills with Two Multifingered Handscited by 72
  4. 2024Bunny-VisionPro: Real-Time Bimanual Dexterous Teleoperation for Imitation Learningcited by 69
  5. 2025HOMIE: Humanoid Loco-Manipulation with Isomorphic Exoskeleton Cockpitcited by 67
  6. 2023Robot Synesthesia: In-Hand Manipulation with Visuotactile Sensingcited by 52
  7. 2023Zero-Shot Robot Manipulation from Passive Human Videoscited by 50
  8. 2024Bidirectional Decoding: Improving Action Chunking via Guided Test-Time Samplingcited by 50
  9. 2022Learning Generalizable Dexterous Manipulation from Human Grasp Affordancecited by 46
  10. 2025VidBot: Learning Generalizable 3D Actions from In-the-Wild 2D Human Videos for Zero-Shot Robotic Manipulationcited by 46
  11. 2024DynaMo: In-Domain Dynamics Pretraining for Visuo-Motor Controlcited by 44
  12. 2023Sequential Dexterity: Chaining Dexterous Policies for Long-Horizon Manipulationcited by 43
  13. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  14. 2024Hand-Object Interaction Pretraining from Videoscited by 38
  15. 2022Learning Continuous Grasping Function with a Dexterous Hand from Human Demonstrationscited by 34
  16. 2025DexWild: Dexterous Human Interactions for In-the-Wild Robot Policiescited by 34
  17. 2025Robotic Manipulation by Imitating Generated Videos Without Physical Demonstrationscited by 34
  18. 2022Learning to Imitate Object Interactions from Internet Videoscited by 32
  19. 2025BEHAVIOR Robot Suite: Streamlining Real-World Whole-Body Manipulation for Everyday Household Activitiescited by 29
  20. 2024TeleMoMa: A Modular and Versatile Teleoperation System for Mobile Manipulationcited by 27
  21. 2024Cross-Embodiment Dexterous Grasping with Reinforcement Learningcited by 26
  22. 2025DexGrasp Anything: Towards Universal Robotic Dexterous Grasping with Physics Awarenesscited by 26
  23. 2025X-Sim: Cross-Embodiment Learning via Real-to-Sim-to-Realcited by 25
  24. 2023DexDeform: Dexterous Deformable Object Manipulation with Human Demonstrations and Differentiable Physicscited by 23
  25. 2024Bimanual Dexterity for Complex Taskscited by 23
  26. 2025Dexterous Manipulation through Imitation Learning: A Surveycited by 20
  27. 2023ContactArt: Learning 3D Interaction Priors for Category-level Articulated Object and Hand Poses Estimationcited by 16
  28. 2024ScrewMimic: Bimanual Imitation from Human Videos with Screw Space Projectioncited by 16
  29. 2023DEFT: Dexterous Fine-Tuning for Real-World Hand Policiescited by 15
  30. 2024DexHandDiff: Interaction-aware Diffusion Planning for Adaptive Dexterous Manipulationcited by 15
  31. 2025Pixel Motion as Universal Representation for Robot Controlcited by 15
  32. 2024Learning Diverse Bimanual Dexterous Manipulation Skills from Human Demonstrationscited by 12
  33. 2025Geometric Retargeting: A Principled, Ultrafast Neural Hand Retargeting Algorithmcited by 12
  34. 2024One-Shot Imitation under Mismatched Executioncited by 11
  35. 2026SimToolReal: An Object-Centric Policy for Zero-Shot Dexterous Tool Manipulationcited by 11
  36. 2023Learning a Universal Human Prior for Dexterous Manipulation from Human Preferencecited by 10
  37. 2024Learning by Watching: A Review of Video-based Learning Approaches for Robot Manipulationcited by 10
  38. 2025Learning to Transfer Human Hand Skills for Robot Manipulationscited by 10

50 most cited of 79 All 79 (JSON)