mlsanityData to 7 Oct 2026

Third-Person Visual Imitation Learning via Decoupled Hierarchical Controller

arXiv 1911.09676 ↗First posted 21 Nov 2019cs.LG cs.CV cs.ROcited by 73

On the map: Offline RL and imitation learning from data → Robot manipulation imitation learning

Abstract (arXiv)

We study a generalized setup for learning from demonstration to build an agent that can manipulate novel objects in unseen scenarios by looking at only a single video of human demonstration from a third-person perspective. To accomplish this goal, our agent should not only learn to understand the intent of the demonstrated third-person video in its context but also perform the intended task in its environment configuration. Our central insight is to enforce this structure explicitly during learning by decoupling what to achieve (intended task) from how to perform it (controller). We propose a hierarchical setup where a high-level module learns to generate a series of first-person sub-goals conditioned on the third-person video demonstration, and a low-level controller predicts the actions to achieve those sub-goals. Our agent acts from raw image observations without any access to the full state information. We show results on a real robotic platform using Baxter for the manipulation tasks of pouring and placing objects in a box. Project video and code are at https://pathak22.github.io/hierarchical-imitation/

Key facts. Cited by 73 papers in the map (data through 7 Oct 2026). Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 23 of its 33 references that are in the map. First posted on arXiv 21 Nov 2019.

Builds on

23 of 33 references.

Closest ancestors · 1

Same direction, newest first

  1. 2018Zero-Shot Visual Imitation23 Apr

Other references · 22

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  5. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  6. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  7. 2014Conditional Generative Adversarial Netscited by 2,152
  8. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  9. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  10. 2018Learning Dexterous In-Hand Manipulationcited by 643
  11. 2014Deep Domain Confusion: Maximizing for Domain Invariancecited by 597
  12. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 453
10 more
  1. 2017Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperationcited by 286
  2. 2016Deep Visual Foresight for Planning Robot Motioncited by 276
  3. 2015Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Imagescited by 266
  4. 2016CAD2RL: Real Single-Image Flight without a Single Real Imagecited by 227
  5. 2018One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learningcited by 146
  6. 2017Self-Supervised Visual Planning with Temporal Skip Connectionscited by 137
  7. 2016Learning Dense Correspondence via 3D-guided Cycle Consistencycited by 91
  8. 2018Multiple Interactions Made Easy (MIME): Large Scale Demonstrations Data for Imitationcited by 76
  9. 2016Unsupervised Perceptual Rewards for Imitation Learningcited by 64
  10. 2016First-Person Activity Forecasting with Online Inverse Reinforcement Learningcited by 40

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 9; 2022: 13; 2023: 15; 2024: 18; 2025: 15; 2026: 2.

73 papers.

Sort:

By area:

  1. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  2. 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 221
  3. 2023MimicPlay: Long-Horizon Imitation Learning by Watching Human Playcited by 214
  4. 2021DexMV: Imitation Learning for Dexterous Manipulation from Human Videoscited by 194
  5. 2024HumanPlus: Humanoid Shadowing and Imitation from Humanscited by 164
  6. 2023Learning to Act from Actionless Videos through Dense Correspondencescited by 159
  7. 2022Human-to-Robot Imitation in the Wildcited by 157
  8. 2023Structured World Models from Human Videoscited by 144
  9. 2024Flow as the Cross-Domain Manipulation Interfacecited by 126
  10. 2021Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videoscited by 116
  11. 2021XIRL: Cross-embodiment Inverse Reinforcement Learningcited by 101
  12. 2022VideoDex: Learning Dexterity from Internet Videoscited by 101
38 more
  1. 2021Learning by Watching: Physical Imitation of Manipulation Skills from Human Videoscited by 87
  2. 2022DexVIP: Learning Dexterous Grasping with Human Hand Pose Priors from Videocited by 83
  3. 2023XSkill: Cross Embodiment Skill Discoverycited by 74
  4. 2025Motion Tracks: A Unified Representation for Human-Robot Transfer in Few-Shot Imitation Learningcited by 57
  5. 2023Zero-Shot Robot Manipulation from Passive Human Videoscited by 49
  6. 2024Vid2Robot: End-to-end Video-conditioned Policy Learning with Cross-Attention Transformerscited by 47
  7. 2024Vision-based Manipulation from Single Human Video with Open-World Object Graphscited by 46
  8. 2021Cross-domain Imitation from Observationscited by 42
  9. 2025Robotic Manipulation by Imitating Generated Videos Without Physical Demonstrationscited by 33
  10. 2023MimicTouch: Leveraging Multi-modal Human Tactile Demonstrations for Contact-rich Manipulationcited by 30
  11. 2022K-VIL: Keypoints-based Visual Imitation Learningcited by 27
  12. 2022Semi-Supervised Offline Reinforcement Learning with Action-Free Trajectoriescited by 27
  13. 2024VideoAgent: Self-Improving Video Generationcited by 24
  14. 2025EgoExoBench: A Benchmark for First- and Third-person View Video Understanding in MLLMscited by 22
  15. 2025Dream2Flow: Bridging Video Generation and Open-World Manipulation with 3D Object Flowcited by 21
  16. 2021Domain-Robust Visual Imitation Learning with Mutual Information Constraintscited by 20
  17. 2021Adversarial Option-Aware Hierarchical Imitation Learningcited by 19
  18. 2024Play to the Score: Stage-Guided Dynamic Multi-Sensory Fusion for Robotic Manipulationcited by 19
  19. 2025Robot Learning from a Physical World Modelcited by 18
  20. 2022Cross-Domain Transfer via Semantic Skill Imitationcited by 17
  21. 2024A Comprehensive Survey of Cross-Domain Policy Transfer for Embodied Agentscited by 16
  22. 2023Hierarchical Imitation Learning with Vector Quantized Modelscited by 15
  23. 2023Learning Video-Conditioned Policies for Unseen Manipulation Taskscited by 15
  24. 2023MoVie: Visual Model-Based Policy Adaptation for View Generalizationcited by 15
  25. 2023Invariant Causal Imitation Learning for Generalizable Policiescited by 15
  26. 2025Pixel Motion as Universal Representation for Robot Controlcited by 15
  27. 2023Domain Adaptive Imitation Learning with Visual Observationcited by 11
  28. 2020A Geometric Perspective on Visual Imitation Learningcited by 10
  29. 2024Learning by Watching: A Review of Video-based Learning Approaches for Robot Manipulationcited by 10
  30. 2024POV: Prompt-Oriented View-Agnostic Learning for Egocentric Hand-Object Interaction in the Multi-View Worldcited by 10
  31. 2024VIEW: Visual Imitation Learning with Waypointscited by 10
  32. 2024Adapt2Reward: Adapting Video-Language Models to Generalizable Robotic Rewards via Failure Promptscited by 9
  33. 2024xTED: Cross-Domain Adaptation via Diffusion-Based Trajectory Editingcited by 9
  34. 2022HERD: Continuous Human-to-Robot Evolution for Learning from Human Demonstrationcited by 8
  35. 2024VLMimic: Vision Language Models are Visual Imitation Learner for Fine-grained Actionscited by 8
  36. 2024Pre-trained Visual Dynamics Representations for Efficient Policy Learningcited by 8
  37. 2025Tool-as-Interface: Learning Robot Policies from Observing Human Tool Usecited by 8
  38. 2025Bridging Perspectives: A Survey on Cross-view Collaborative Intelligence with Egocentric-Exocentric Visioncited by 8

50 most cited of 73 All 73 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2018One-Shot Hierarchical Imitation Learning of Compound Visuomotor Tasks
  2. 2023Active Third-Person Imitation Learning
  3. 2021Coarse-to-Fine Imitation Learning: Robot Manipulation from a Single Demonstration
  4. 2026Human-to-Robot Interaction: Learning from Video Demonstration for Robot Imitation
  5. 2021Manipulator-Independent Representations for Visual Imitation
  6. 2026Mirror Learning
  7. 2021The Surprising Effectiveness of Representation Learning for Visual Imitation
  8. 2019Learning One-Shot Imitation from Humans without Humans