mlsanityData to 14 Oct 2026

Human-to-Robot Imitation in the Wild

arXiv 2207.09450 ↗First posted 19 Jul 2022cs.RO cs.AI cs.CVcited by 161

Not on the map. Why?

Abstract (arXiv)

We approach the problem of learning by watching humans in the wild. While traditional approaches in Imitation and Reinforcement Learning are promising for learning in the real world, they are either sample inefficient or are constrained to lab settings. Meanwhile, there has been a lot of success in processing passive, unstructured human data. We propose tackling this problem via an efficient one-shot robot learning algorithm, centered around learning from a third-person perspective. We call our method WHIRL: In-the-Wild Human Imitating Robot Learning. WHIRL extracts a prior over the intent of the human demonstrator, using it to initialize our agent's policy. We introduce an efficient real-world policy learning scheme that improves using interactions. Our key contributions are a simple sampling-based policy optimization approach, a novel objective function for aligning human and robot videos as well as an exploration method to boost sample efficiency. We show one-shot generalization and success in real-world settings, including 20 different manipulation tasks in the wild. Videos and talk at https://human2robot.github.io

Key facts. Cited by 161 papers in the map (data through 14 Oct 2026). Builds on 56 of its 87 references that are in the map. First posted on arXiv 19 Jul 2022.

Builds on

56 of 87 references.

References · 56

  1. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  3. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  6. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  7. 2017Non-local Neural Networkscited by 2,293
  8. 2018SlowFast Networks for Video Recognitioncited by 1,867
  9. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  10. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  11. 2017Hindsight Experience Replaycited by 1,136
  12. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,103
38 more
  1. 2021Video Swin Transformercited by 1,091
  2. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 998
  3. 2017End-to-end Recovery of Human Shape and Posecited by 885
  4. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 856
  5. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 795
  6. 2019Solving Rubik's Cube with a Robot Handcited by 734
  7. 2017Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?cited by 551
  8. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 459
  9. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  10. 2018Residual Reinforcement Learning for Robot Controlcited by 274
  11. 2017Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metricscited by 264
  12. 2017One-Shot Imitation Learningcited by 233
  13. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 229
  14. 2017One-Shot Visual Imitation Learning via Meta-Learningcited by 228
  15. 2019Skew-Fit: State-Covering Self-Supervised Reinforcement Learningcited by 174
  16. 2021FrankMocap: A Monocular 3D Whole-Body Pose Estimation System via Regression and Integrationcited by 169
  17. 2017Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translationcited by 167
  18. 2019Temporal Cycle-Consistency Learningcited by 154
  19. 2021VideoMoCo: Contrastive Video Representation Learning with Temporally Adversarial Examplescited by 154
  20. 2021The Surprising Effectiveness of Representation Learning for Visual Imitationcited by 154
  21. 2018One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learningcited by 148
  22. 2019ContactDB: Analyzing and Predicting Grasp Contact via Thermal Imagingcited by 147
  23. 2021Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videoscited by 119
  24. 2018Zero-Shot Visual Imitationcited by 117
  25. 2020The Ingredients of Real-World Robotic Reinforcement Learningcited by 117
  26. 2020Visual Imitation Made Easycited by 111
  27. 2019Unsupervised Learning of Object Keypoints for Perception and Controlcited by 109
  28. 2019Grasping in the Wild:Learning 6DoF Closed-Loop Grasping from Low-Cost Demonstrationscited by 107
  29. 2021XIRL: Cross-embodiment Inverse Reinforcement Learningcited by 107
  30. 2019AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videoscited by 105
  31. 2021Learning by Watching: Physical Imitation of Manipulation Skills from Human Videoscited by 87
  32. 2018Multiple Interactions Made Easy (MIME): Large Scale Demonstrations Data for Imitationcited by 76
  33. 2018Robot Learning in Homes: Improving Generalization and Reducing Dataset Biascited by 75
  34. 2019Third-Person Visual Imitation Learning via Decoupled Hierarchical Controllercited by 73
  35. 2016Unsupervised Perceptual Rewards for Imitation Learningcited by 65
  36. 2019Copy-and-Paste Networks for Deep Video Inpaintingcited by 58
  37. 2017EX2: Exploration with Exemplar Models for Deep Reinforcement Learningcited by 42
  38. 2021The Design of Stretch: A Compact, Lightweight Mobile Manipulator for Indoor Human Environmentscited by 40

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 6; 2023: 31; 2024: 55; 2025: 46; 2026: 23.

161 papers.

Sort:

By area:

  1. 2024Octo: An Open-Source Generalist Robot Policycited by 1,484
  2. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 750
  3. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 395
  4. 2024Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperationcited by 391
  5. 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 226
  6. 2023MimicPlay: Long-Horizon Imitation Learning by Watching Human Playcited by 223
  7. 2024HumanPlus: Humanoid Shadowing and Imitation from Humanscited by 185
  8. 2024EgoMimic: Scaling Imitation Learning via Egocentric Videocited by 175
  9. 2023RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunkingcited by 169
  10. 2023Language-Driven Representation Learning for Roboticscited by 165
  11. 2023Learning to Act from Actionless Videos through Dense Correspondencescited by 164
  12. 2024DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulationcited by 160
38 more
  1. 2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedbackcited by 156
  2. 2024Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulationcited by 153
  3. 2023Structured World Models from Human Videoscited by 147
  4. 2024Open-TeleVision: Teleoperation with Immersive Active Visual Feedbackcited by 145
  5. 2024Data Scaling Laws in Imitation Learning for Robotic Manipulationcited by 145
  6. 2026π0.7: a Steerable Generalist Robotic Foundation Model with Emergent Capabilitiescited by 138
  7. 2023Large Language Models for Robotics: A Surveycited by 133
  8. 2024Flow as the Cross-Domain Manipulation Interfacecited by 132
  9. 2024Track2Act: Predicting Point Tracks from Internet Videos enables Generalizable Robot Manipulationcited by 124
  10. 2023Affordance Diffusion: Synthesizing Hand-Object Interactionscited by 110
  11. 2022VideoDex: Learning Dexterity from Internet Videoscited by 106
  12. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 103
  13. 2023On Bringing Robots Homecited by 101
  14. 2024General Flow as Foundation Affordance for Scalable Robot Learningcited by 85
  15. 2023LEAP Hand: Low-Cost, Efficient, and Anthropomorphic Hand for Robot Learningcited by 79
  16. 2025Humanoid Policy ~ Human Policycited by 78
  17. 2023PhysHOI: Physics-Based Imitation of Dynamic Human-Object Interactioncited by 76
  18. 2023XSkill: Cross Embodiment Skill Discoverycited by 75
  19. 2026Being-H0.7: A Latent World-Action Model from Egocentric Videoscited by 75
  20. 2024Robo-ABC: Affordance Generalization Beyond Categories via Semantic Correspondence for Robot Manipulationcited by 71
  21. 2026Being-H0.5: Scaling Human-Centric Robot Learning for Cross-Embodiment Generalizationcited by 66
  22. 2024RAM: Retrieval-Based Affordance Transfer for Generalizable Zero-Shot Robotic Manipulationcited by 58
  23. 2026EgoVerse: An Egocentric Human Dataset for Robot Learning from Around the Worldcited by 57
  24. 2024AffordanceLLM: Grounding Affordance from Vision Language Modelscited by 55
  25. 2024Robot See Robot Do: Imitating Articulated Object Manipulation with Monocular 4D Reconstructioncited by 54
  26. 2023Towards Generalizable Zero-Shot Manipulation via Translating Human Interaction Planscited by 52
  27. 2024Vid2Robot: End-to-end Video-conditioned Policy Learning with Cross-Attention Transformerscited by 52
  28. 2023Zero-Shot Robot Manipulation from Passive Human Videoscited by 51
  29. 2024Vision-based Manipulation from Single Human Video with Open-World Object Graphscited by 50
  30. 2022Retrospectives on the Embodied AI Workshopcited by 46
  31. 2025VidBot: Learning Generalizable 3D Actions from In-the-Wild 2D Human Videos for Zero-Shot Robotic Manipulationcited by 46
  32. 2024DynaMo: In-Domain Dynamics Pretraining for Visuo-Motor Controlcited by 44
  33. 2025Emergence of Human to Robot Transfer in Vision-Language-Action Modelscited by 42
  34. 2024CityWalker: Learning Embodied Urban Navigation from Web-Scale Videoscited by 41
  35. 2024Hand-Object Interaction Pretraining from Videoscited by 39
  36. 2024R+X: Retrieval and Execution from Everyday Human Videoscited by 38
  37. 2025Robotic Manipulation by Imitating Generated Videos Without Physical Demonstrationscited by 38
  38. 2024VLM See, Robot Do: Human Demo Video to Robot Action Plan via Vision Language Modelcited by 37

50 most cited of 161 All 161 (JSON)