mlsanityData to 7 Oct 2026

Learning by Watching: Physical Imitation of Manipulation Skills from Human Videos

arXiv 2101.07241 ↗First posted 18 Jan 2021cs.RO cs.CV cs.LGcited by 87

Not on the map. Why?

Cited by 22 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Learning from visual data opens the potential to accrue a large range of manipulation behaviors by leveraging human demonstrations without specifying each of them mathematically, but rather through natural task specification. In this paper, we present Learning by Watching (LbW), an algorithmic framework for policy learning through imitation from a single video specifying the task. The key insights of our method are two-fold. First, since the human arms may not have the same morphology as robot arms, our framework learns unsupervised human to robot translation to overcome the morphology mismatch issue. Second, to capture the details in salient regions that are crucial for learning state representations, our model performs unsupervised keypoint detection on the translated robot videos. The detected keypoints form a structured representation that contains semantically meaningful information and can be used directly for computing reward and policy learning. We evaluate the effectiveness of our LbW framework on five robot manipulation tasks, including reaching, pushing, sliding, coffee making, and drawer closing. Extensive experimental evaluations demonstrate that our method performs favorably against the state-of-the-art approaches.

Key facts. Cited by 87 papers in the map (data through 7 Oct 2026). Builds on 45 of its 56 references that are in the map. First posted on arXiv 18 Jan 2021.

Builds on

45 of 56 references.

References · 45

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  3. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  4. 2017Hindsight Experience Replaycited by 1,096
  5. 2017StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translationcited by 1,036
  6. 2018Multimodal Unsupervised Image-to-Image Translationcited by 795
  7. 2017Unsupervised Image-to-Image Translation Networkscited by 602
  8. 2018Behavioral Cloning from Observationcited by 482
  9. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 453
  10. 2017Toward Multimodal Image-to-Image Translationcited by 295
  11. 2017Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperationcited by 286
  12. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 268
33 more
  1. 2019DRIT++: Diverse Image-to-Image Translation via Disentangled Representationscited by 244
  2. 2019Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Modelcited by 228
  3. 2018RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitationcited by 202
  4. 2018Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via Randomized-to-Canonical Adaptation Networkscited by 171
  5. 2017Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translationcited by 167
  6. 2015Deep Spatial Autoencoders for Visuomotor Learningcited by 149
  7. 2019Seeing What a GAN Cannot Generatecited by 148
  8. 2018One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learningcited by 146
  9. 2018State Representation Learning for Control: An Overviewcited by 125
  10. 2018Zero-Shot Visual Imitationcited by 114
  11. 2020Learning to Generalize Across Long-Horizon Tasks from Human Demonstrationscited by 114
  12. 2018Unsupervised Learning of Object Landmarks through Conditional Image Generationcited by 113
  13. 2019Unsupervised Learning of Object Keypoints for Perception and Controlcited by 109
  14. 2020CrDoCo: Pixel-level Domain Transfer with Cross-Domain Consistencycited by 107
  15. 2019AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videoscited by 105
  16. 2020Human-in-the-Loop Imitation Learning using Remote Teleoperationcited by 103
  17. 2019Unsupervised Learning of Object Structure and Dynamics from Videoscited by 92
  18. 2020RL-CycleGAN: Reinforcement Learning Aware Simulation-To-Realcited by 89
  19. 2019IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Datacited by 85
  20. 2020Reinforcement Learning with Videos: Combining Offline Observations with Interactioncited by 80
  21. 2018Unsupervised Discovery of Object Landmarks as Structural Representationscited by 73
  22. 2019Third-Person Visual Imitation Learning via Decoupled Hierarchical Controllercited by 73
  23. 2020Keypoints into the Future: Self-Supervised Correspondence in Model-Based Reinforcement Learningcited by 69
  24. 2018Variational Inverse Control with Events: A General Framework for Data-Driven Reward Definitioncited by 66
  25. 2016Unsupervised Perceptual Rewards for Imitation Learningcited by 64
  26. 2017Unsupervised learning of object landmarks by factorized spatial embeddingscited by 64
  27. 2018Deforming Autoencoders: Unsupervised Disentangling of Shape and Appearancecited by 58
  28. 2018Transfer Learning for Related Reinforcement Learning Tasks via Image-to-Image Translationcited by 50
  29. 2018Self-supervised learning of a facial attribute embedding from videocited by 44
  30. 2019Graph-Structured Visual Imitationcited by 43
  31. 2020Learning Cross-Domain Correspondence for Control with Dynamics Cycle-Consistencycited by 43
  32. 2018SFV: Reinforcement Learning of Physical Skills from Videoscited by 20
  33. 2020Learning Object Manipulation Skills via Approximate State Estimation from Real Videoscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 13; 2023: 12; 2024: 25; 2025: 23; 2026: 11.

87 papers.

Sort:

By area:

  1. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  2. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 384
  3. 2024Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperationcited by 377
  4. 2025Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasetscited by 218
  5. 2023MimicPlay: Long-Horizon Imitation Learning by Watching Human Playcited by 214
  6. 2023Any-point Trajectory Modeling for Policy Learningcited by 207
  7. 2021DexMV: Imitation Learning for Dexterous Manipulation from Human Videoscited by 194
  8. 2024HumanPlus: Humanoid Shadowing and Imitation from Humanscited by 164
  9. 2024EgoMimic: Scaling Imitation Learning via Egocentric Videocited by 160
  10. 2022Human-to-Robot Imitation in the Wildcited by 157
  11. 2023Structured World Models from Human Videoscited by 144
  12. 2024Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulationcited by 142
38 more
  1. 2024Flow as the Cross-Domain Manipulation Interfacecited by 126
  2. 2021Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videoscited by 116
  3. 2024Track2Act: Predicting Point Tracks from Internet Videos enables Generalizable Robot Manipulationcited by 116
  4. 2022You Only Demonstrate Once: Category-Level Manipulation from Single Visual Demonstrationcited by 79
  5. 2022From One Hand to Multiple Hands: Imitation Learning for Dexterous Manipulation from Single-Camera Teleoperationcited by 79
  6. 2023XSkill: Cross Embodiment Skill Discoverycited by 74
  7. 2025Large Video Planner Enables Generalizable Robot Controlcited by 67
  8. 2025Humanoid Policy ~ Human Policycited by 65
  9. 2025Scalable Vision-Language-Action Model Pretraining for Robotic Manipulation with Real-Life Human Activity Videoscited by 63
  10. 2023Towards Generalizable Zero-Shot Manipulation via Translating Human Interaction Planscited by 51
  11. 2023Zero-Shot Robot Manipulation from Passive Human Videoscited by 49
  12. 2026EgoVerse: An Egocentric Human Dataset for Robot Learning from Around the Worldcited by 49
  13. 2024Vision-based Manipulation from Single Human Video with Open-World Object Graphscited by 46
  14. 2025VidBot: Learning Generalizable 3D Actions from In-the-Wild 2D Human Videos for Zero-Shot Robotic Manipulationcited by 46
  15. 2022Graph Inverse Reinforcement Learning from Diverse Videoscited by 45
  16. 2021Articulated Object Interaction in Unknown Scenes with Whole-Body Mobile Manipulationcited by 44
  17. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  18. 2024Surgical Robot Transformer (SRT): Imitation Learning for Surgical Taskscited by 38
  19. 2024Hand-Object Interaction Pretraining from Videoscited by 38
  20. 2024ViViDex: Learning Vision-based Dexterous Manipulation from Human Videoscited by 33
  21. 2023MimicTouch: Leveraging Multi-modal Human Tactile Demonstrations for Contact-rich Manipulationcited by 30
  22. 2023AR2-D2:Training a Robot Without a Robotcited by 26
  23. 2024Learning an Actionable Discrete Diffusion Policy via Large-Scale Actionless Video Pre-Trainingcited by 26
  24. 2024Mitigating the Human-Robot Domain Discrepancy in Visual Pre-training for Robotic Manipulationcited by 22
  25. 2023Look Ma, No Hands! Agent-Environment Factorization of Egocentric Videoscited by 21
  26. 2025In-N-On: Scaling Egocentric Manipulation with in-the-wild and on-task Datacited by 21
  27. 2025AMPLIFY: Actionless Motion Priors for Robot Learning from Videoscited by 18
  28. 2025Robot Learning from a Physical World Modelcited by 18
  29. 2025AirExo-2: Scaling up Generalizable Robotic Imitation Learning with Low-Cost Exoskeletonscited by 17
  30. 2024ScrewMimic: Bimanual Imitation from Human Videos with Screw Space Projectioncited by 16
  31. 2023Learning Video-Conditioned Policies for Unseen Manipulation Taskscited by 15
  32. 2026RoboWM-Bench: A Benchmark for Evaluating World Models in Robotic Manipulationcited by 15
  33. 2022Learning Reward Functions for Robotic Manipulation by Observing Humanscited by 14
  34. 2026HumanEgo: Zero-Shot Robot Learning from Minutes of Human Egocentric Videoscited by 14
  35. 2025Object-centric 3D Motion Field for Robot Learning from Human Videoscited by 13
  36. 2024One-Shot Imitation under Mismatched Executioncited by 11
  37. 2024ILBiT: Imitation Learning for Robot Using Position and Torque Information based on Bilateral Control with Transformercited by 10
  38. 2024Learning by Watching: A Review of Video-based Learning Approaches for Robot Manipulationcited by 10

50 most cited of 87 All 87 (JSON)