mlsanityData to 7 Oct 2026

VideoDex: Learning Dexterity from Internet Videos

arXiv 2212.04498 ↗First posted 8 Dec 2022cs.RO cs.AI cs.CVcited by 101

Not on the map. Why?

Abstract (arXiv)

To build general robotic agents that can operate in many environments, it is often imperative for the robot to collect experience in the real world. However, this is often not feasible due to safety, time, and hardware restrictions. We thus propose leveraging the next best thing as real-world experience: internet videos of humans using their hands. Visual priors, such as visual features, are often learned from videos, but we believe that more information from videos can be utilized as a stronger prior. We build a learning algorithm, VideoDex, that leverages visual, action, and physical priors from human video datasets to guide robot behavior. These actions and physical priors in the neural network dictate the typical human behavior for a particular robot task. We test our approach on a robot arm and dexterous hand-based system and show strong results on various manipulation tasks, outperforming various state-of-the-art methods. Videos at https://video-dex.github.io

Key facts. Cited by 101 papers in the map (data through 7 Oct 2026). Builds on 47 of its 73 references that are in the map. First posted on arXiv 8 Dec 2022.

Builds on

47 of 73 references.

References · 47

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  7. 2017Mask R-CNNcited by 7,259
  8. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  9. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,600
  10. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  11. 2018OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,161
  12. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
35 more
  1. 2016End to End Learning for Self-Driving Carscited by 1,000
  2. 2017End-to-end Recovery of Human Shape and Posecited by 883
  3. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 882
  4. 2020ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial and Multi-Map SLAMcited by 862
  5. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  6. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  7. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  8. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  9. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 613
  10. 2020AdaBins: Depth Estimation using Adaptive Binscited by 596
  11. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 453
  12. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  13. 2019FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Imagescited by 271
  14. 2018Visual Reinforcement Learning with Imagined Goalscited by 248
  15. 2022Masked Visual Pre-training for Motor Controlcited by 246
  16. 2021DexMV: Imitation Learning for Dexterous Manipulation from Human Videoscited by 194
  17. 2021FrankMocap: A Monocular 3D Whole-Body Pose Estimation System via Regression and Integrationcited by 168
  18. 2022Human-to-Robot Imitation in the Wildcited by 157
  19. 2021The Surprising Effectiveness of Representation Learning for Visual Imitationcited by 149
  20. 2021Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videoscited by 116
  21. 2020Visual Imitation Made Easycited by 111
  22. 2019DexPilot: Vision Based Teleoperation of Dexterous Robotic Hand-Arm Systemcited by 105
  23. 2019AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videoscited by 105
  24. 2022DexVIP: Learning Dexterous Grasping with Human Hand Pose Priors from Videocited by 83
  25. 2020Reinforcement Learning with Videos: Combining Offline Observations with Interactioncited by 80
  26. 2022Robotic Telekinesis: Learning a Robotic Hand Imitator by Watching Humans on Youtubecited by 79
  27. 2022From One Hand to Multiple Hands: Imitation Learning for Dexterous Manipulation from Single-Camera Teleoperationcited by 79
  28. 2022Dexterous Imitation Made Easy: A Learning-Based Framework for Efficient Dexterous Manipulationcited by 75
  29. 2019Third-Person Visual Imitation Learning via Decoupled Hierarchical Controllercited by 73
  30. 2020Neural Dynamic Policies for End-to-End Sensorimotor Learningcited by 56
  31. 2020Model-Based Inverse Reinforcement Learning from Visual Demonstrationscited by 52
  32. 2016The Curious Robot: Learning Visual Representations via Physical Interactionscited by 37
  33. 2022RB2: Robotic Manipulation Benchmarking with a Twistcited by 23
  34. 2018Vision-based Teleoperation of Shadow Dexterous Hand using End-to-End Deep Neural Networkcited by 19
  35. 2021Hierarchical Neural Dynamic Policiescited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 17; 2024: 30; 2025: 34; 2026: 20.

101 papers.

Sort:

By area:

  1. 2024Latent Action Pretraining from Videoscited by 302
  2. 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 221
  3. 2023MimicPlay: Long-Horizon Imitation Learning by Watching Human Playcited by 214
  4. 2023Any-point Trajectory Modeling for Policy Learningcited by 207
  5. 2025DreamGen: Unlocking Generalization in Robot Learning through Video World Modelscited by 168
  6. 2023RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunkingcited by 167
  7. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
  8. 2024DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulationcited by 149
  9. 2023Structured World Models from Human Videoscited by 144
  10. 2024Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulationcited by 142
  11. 2024Open-TeleVision: Teleoperation with Immersive Active Visual Feedbackcited by 137
  12. 2023AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation Systemcited by 136
38 more
  1. 2025mimic-video: Video-Action Models for Generalizable Robot Control Beyond VLAscited by 133
  2. 2026π0.7: a Steerable Generalist Robotic Foundation Model with Emergent Capabilitiescited by 127
  3. 2024Flow as the Cross-Domain Manipulation Interfacecited by 126
  4. 2024Track2Act: Predicting Point Tracks from Internet Videos enables Generalizable Robot Manipulationcited by 116
  5. 2025FLARE: Robot Learning with Implicit World Modelingcited by 105
  6. 2023An Outlook into the Future of Egocentric Visioncited by 84
  7. 2023LEAP Hand: Low-Cost, Efficient, and Anthropomorphic Hand for Robot Learningcited by 78
  8. 2025Scalable Vision-Language-Action Model Pretraining for Robotic Manipulation with Real-Life Human Activity Videoscited by 63
  9. 2025ManipTrans: Efficient Dexterous Bimanual Manipulation Transfer via Residual Learningcited by 56
  10. 2023Towards Generalizable Zero-Shot Manipulation via Translating Human Interaction Planscited by 51
  11. 2023Zero-Shot Robot Manipulation from Passive Human Videoscited by 49
  12. 2025VidBot: Learning Generalizable 3D Actions from In-the-Wild 2D Human Videos for Zero-Shot Robotic Manipulationcited by 46
  13. 2024PoCo: Policy Composition from and for Heterogeneous Robot Learningcited by 42
  14. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  15. 2024HRP: Human Affordances for Robotic Pre-Trainingcited by 40
  16. 2024Hand-Object Interaction Pretraining from Videoscited by 38
  17. 2023Dexterous Functional Graspingcited by 36
  18. 2024SafeSora: Towards Safety Alignment of Text2Video Generation via a Human Preference Datasetcited by 34
  19. 2025DexWild: Dexterous Human Interactions for In-the-Wild Robot Policiescited by 34
  20. 2024ViViDex: Learning Vision-based Dexterous Manipulation from Human Videoscited by 33
  21. 2024ACE: A Cross-Platform Visual-Exoskeletons System for Low-Cost Dexterous Teleoperationcited by 32
  22. 2024ARCap: Collecting High-quality Human Demonstrations for Robot Learning with Augmented Reality Feedbackcited by 31
  23. 2025ZeroMimic: Distilling Robotic Manipulation Skills from Web Videoscited by 31
  24. 2025DexMachina: Functional Retargeting for Bimanual Dexterous Manipulationcited by 30
  25. 2023AR2-D2:Training a Robot Without a Robotcited by 26
  26. 2024Learning an Actionable Discrete Diffusion Policy via Large-Scale Actionless Video Pre-Trainingcited by 26
  27. 2025World Models for Learning Dexterous Hand-Object Interactions from Human Videoscited by 26
  28. 2024CyberDemo: Augmenting Simulated Human Demonstration for Real-World Dexterous Manipulationcited by 25
  29. 2025DexGrasp Anything: Towards Universal Robotic Dexterous Grasping with Physics Awarenesscited by 25
  30. 2025X-Sim: Cross-Embodiment Learning via Real-to-Sim-to-Realcited by 25
  31. 2025SPIDER: Scalable Physics-Informed Dexterous Retargetingcited by 25
  32. 2023Transferring Foundation Models for Generalizable Robotic Manipulationcited by 18
  33. 2025Dexterous Manipulation through Imitation Learning: A Surveycited by 18
  34. 2025AMPLIFY: Actionless Motion Priors for Robot Learning from Videoscited by 18
  35. 2024Semantically Controllable Augmentations for Generalizable Robot Learningcited by 17
  36. 2024ScrewMimic: Bimanual Imitation from Human Videos with Screw Space Projectioncited by 16
  37. 2025DemoDiffusion: One-Shot Human Imitation using pre-trained Diffusion Policycited by 16
  38. 2025DEXOP: A Device for Robotic Transfer of Dexterous Human Manipulationcited by 15

50 most cited of 101 All 101 (JSON)