mlsanityData to 7 Oct 2026

Zero-Shot Robot Manipulation from Passive Human Videos

arXiv 2302.02011 ↗First posted 3 Feb 2023cs.RO cs.LGcited by 49

Not on the map. Why?

Abstract (arXiv)

Can we learn robot manipulation for everyday tasks, only by watching videos of humans doing arbitrary tasks in different unstructured settings? Unlike widely adopted strategies of learning task-specific behaviors or direct imitation of a human video, we develop a a framework for extracting agent-agnostic action representations from human videos, and then map it to the agent's embodiment during deployment. Our framework is based on predicting plausible human hand trajectories given an initial image of a scene. After training this prediction model on a diverse set of human videos from the internet, we deploy the trained model zero-shot for physical robot manipulation tasks, after appropriate transformations to the robot's embodiment. This simple strategy lets us solve coarse manipulation tasks like opening and closing drawers, pushing, and tool use, without access to any in-domain robot manipulation trajectories. Our real-world deployment results establish a strong baseline for action prediction information that can be acquired from diverse arbitrary videos of human activities, and be useful for zero-shot robotic manipulation in unseen scenes.

Key facts. Cited by 49 papers in the map (data through 7 Oct 2026). Builds on 62 of its 75 references that are in the map. First posted on arXiv 3 Feb 2023.

Builds on

62 of 75 references.

References · 62

  1. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  2. 2016Generative Adversarial Imitation Learningcited by 1,576
  3. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  4. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  5. 2016End to End Learning for Self-Driving Carscited by 1,000
  6. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 899
  7. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  8. 2017PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenescited by 768
  9. 2022Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Itemscited by 601
  10. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  11. 2018Behavioral Cloning from Observationcited by 482
  12. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 453
38 more
  1. 2019Learning joint reconstruction of hands and manipulated objectscited by 377
  2. 2021DexYCB: A Benchmark for Capturing Hand Grasping of Objectscited by 320
  3. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  4. 2017SSD-6D: Making RGB-based 3D detection and 6D pose estimation great againcited by 243
  5. 2017Learning to Estimate 3D Hand Pose from Single RGB Imagescited by 234
  6. 2017First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotationscited by 229
  7. 2020Learning Agile Robotic Locomotion Skills by Imitating Animalscited by 229
  8. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 226
  9. 2017BB8: A Scalable, Accurate, Robust to Partial Occlusion Method for Predicting the 3D Poses of Challenging Objects without Using Depthcited by 225
  10. 2017One-Shot Visual Imitation Learning via Meta-Learningcited by 225
  11. 2019PVN3D: A Deep Point-wise 3D Keypoints Voting Network for 6DoF Pose Estimationcited by 204
  12. 2018RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitationcited by 202
  13. 2021DexMV: Imitation Learning for Dexterous Manipulation from Human Videoscited by 194
  14. 20193D Hand Shape and Pose from Images in the Wildcited by 190
  15. 2018Generative Adversarial Imitation from Observationcited by 183
  16. 2021Where2Act: From Pixels to Actions for Articulated 3D Objectscited by 179
  17. 20193D Hand Shape and Pose Estimation from a Single RGB Imagecited by 174
  18. 2022Human-to-Robot Imitation in the Wildcited by 157
  19. 2020Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wildcited by 146
  20. 2021Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Timecited by 140
  21. 2018Grounded Human-Object Interaction Hotspots from Videocited by 128
  22. 2020RAFT-3D: Scene Flow using Rigid-Motion Embeddingscited by 121
  23. 2019Pushing the Envelope for RGB-based Dense 3D Hand Pose Estimation via Neural Renderingcited by 115
  24. 2018Zero-Shot Visual Imitationcited by 114
  25. 2018Hand Pose Estimation via Latent 2.5D Heatmap Regressioncited by 111
  26. 2020Visual Imitation Made Easycited by 111
  27. 2022Joint Hand Motion and Interaction Hotspots Prediction from Egocentric Videoscited by 110
  28. 2019AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videoscited by 105
  29. 2018Segmentation-driven 6D Object Pose Estimationcited by 104
  30. 2019What Would You Expect? Anticipating Egocentric Actions with Rolling-Unrolling LSTMs and Modality Attentioncited by 104
  31. 2019Grasping in the Wild:Learning 6DoF Closed-Loop Grasping from Low-Cost Demonstrationscited by 104
  32. 2022VideoDex: Learning Dexterity from Internet Videoscited by 101
  33. 2015Benchmarking in Manipulation Research: The YCB Object and Model Set and Benchmarking Protocolscited by 98
  34. 2018Playing hard exploration games by watching YouTubecited by 98
  35. 2022What's in your hands? 3D Reconstruction of Generic Objects in Handscited by 90
  36. 2018Cross-modal Deep Variational Hand Pose Estimationcited by 89
  37. 2019ContactGrasp: Functional Multi-finger Grasp Synthesis from Contactcited by 87
  38. 2021Learning by Watching: Physical Imitation of Manipulation Skills from Human Videoscited by 87

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 17; 2025: 15; 2026: 9.

49 papers.

Sort:

By area:

  1. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  2. 2024Latent Action Pretraining from Videoscited by 302
  3. 2023Any-point Trajectory Modeling for Policy Learningcited by 207
  4. 2025DreamGen: Unlocking Generalization in Robot Learning through Video World Modelscited by 168
  5. 2023RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunkingcited by 167
  6. 2023Learning to Act from Actionless Videos through Dense Correspondencescited by 159
  7. 2026π0.7: a Steerable Generalist Robotic Foundation Model with Emergent Capabilitiescited by 127
  8. 2025FLARE: Robot Learning with Implicit World Modelingcited by 105
  9. 2024General Flow as Foundation Affordance for Scalable Robot Learningcited by 80
  10. 2023Towards Generalizable Zero-Shot Manipulation via Translating Human Interaction Planscited by 51
  11. 2024Vision-based Manipulation from Single Human Video with Open-World Object Graphscited by 46
  12. 2025VidBot: Learning Generalizable 3D Actions from In-the-Wild 2D Human Videos for Zero-Shot Robotic Manipulationcited by 46
37 more
  1. 2024Hand-Object Interaction Pretraining from Videoscited by 38
  2. 2025Robotic Manipulation by Imitating Generated Videos Without Physical Demonstrationscited by 33
  3. 2024Put Myself in Your Shoes: Lifting the Egocentric Perspective from Exocentric Videoscited by 32
  4. 2025ZeroMimic: Distilling Robotic Manipulation Skills from Web Videoscited by 31
  5. 2023MimicTouch: Leveraging Multi-modal Human Tactile Demonstrations for Contact-rich Manipulationcited by 30
  6. 2024DITTO: Demonstration Imitation by Trajectory Transformationcited by 28
  7. 2024Closing the Gap between TD Learning and Supervised Learning -- A Generalisation Point of Viewcited by 26
  8. 2024Learning an Actionable Discrete Diffusion Policy via Large-Scale Actionless Video Pre-Trainingcited by 26
  9. 2025X-Sim: Cross-Embodiment Learning via Real-to-Sim-to-Realcited by 25
  10. 2025Guiding Human-Object Interactions with Rich Geometry and Relationscited by 24
  11. 2025ViPRA: Video Prediction for Robot Actionscited by 24
  12. 2024Ag2Manip: Learning Novel Manipulation Skills with Agent-Agnostic Visual and Action Representationscited by 23
  13. 2025EgoExo-Gen: Ego-centric Video Prediction by Watching Exo-centric Videoscited by 19
  14. 2025AMPLIFY: Actionless Motion Priors for Robot Learning from Videoscited by 18
  15. 2025Robot Learning from a Physical World Modelcited by 18
  16. 2024Semantically Controllable Augmentations for Generalizable Robot Learningcited by 17
  17. 2025MAPLE: Encoding Dexterous Robotic Manipulation Priors Learned From Egocentric Videoscited by 12
  18. 2025A Survey on Imitation Learning for Contact-Rich Tasks in Roboticscited by 12
  19. 2023InteRACT: Transformer Models for Human Intent Prediction Conditioned on Robot Actionscited by 10
  20. 2024Learning by Watching: A Review of Video-based Learning Approaches for Robot Manipulationcited by 10
  21. 2024On-Robot Reinforcement Learning with Goal-Contrastive Rewardscited by 7
  22. 2024Bidirectional Progressive Transformer for Interaction Intention Anticipationcited by 6
  23. 2025X-Diffusion: Training Diffusion Policies on Cross-Embodiment Human Demonstrationscited by 6
  24. 2026WAM-TTT: Steering World-Action Models by Watching Human Play at Test Timecited by 6
  25. 2024TieBot: Learning to Knot a Tie from Visual Demonstration through a Real-to-Sim-to-Real Approachcited by 5
  26. 2026Beyond Data Scaling: Representation-Centric Continued Pre-training for Vision-Language-Action Modelscited by 5
  27. 2024One-shot Video Imitation via Parameterized Symbolic Abstraction Graphscited by 3
  28. 2024A Minimalist Prompt for Zero-Shot Policy Learningcited by 2
  29. 2026ViHOI: Human-Object Interaction Synthesis with Visual Priorscited by 2
  30. 2023Spacewalk-18: A Benchmark for Multimodal and Long-form Procedural Video Understanding in Novel Domainscited by 1
  31. 2026Imitating What Works: Simulation-Filtered Modular Policy Learning from Human Videoscited by 1
  32. 2026Imitation from Heterogeneous Demonstrations using Grounded Latent-Action World Modelscited by 1
  33. 2024PACA: Perspective-Aware Cross-Attention Representation for Zero-Shot Scene Rearrangementcited by 0
  34. 2025OpenTie: Open-vocabulary Sequential Rebar Tying Systemcited by 0
  35. 2026Flow-Enabled Generalization to Human Demonstrations in Few-Shot Imitation Learningcited by 0
  36. 2026MaMi-HOI: Harmonizing Global Kinematics and Local Geometry for Human-Object Interaction Generationcited by 0
  37. 2026Triplet2Track: A Hierarchical System with Object-Centric Representations for Reliable Long-Horizon Manipulationcited by 0