mlsanityData to 7 Oct 2026

Coarse-to-Fine Imitation Learning: Robot Manipulation from a Single Demonstration

arXiv 2105.06411 ↗First posted 13 May 2021cs.RO cs.LGcited by 92

On the map: Offline RL and imitation learning from data → Robot manipulation imitation learning

Abstract (arXiv)

We introduce a simple new method for visual imitation learning, which allows a novel robot manipulation task to be learned from a single human demonstration, without requiring any prior knowledge of the object being interacted with. Our method models imitation learning as a state estimation problem, with the state defined as the end-effector's pose at the point where object interaction begins, as observed from the demonstration. By then modelling a manipulation task as a coarse, approach trajectory followed by a fine, interaction trajectory, this state estimator can be trained in a self-supervised manner, by automatically moving the end-effector's camera around the object. At test time, the end-effector moves to the estimated state through a linear path, at which point the original demonstration's end-effector velocities are simply replayed. This enables convenient acquisition of a complex interaction trajectory, without actually needing to explicitly learn a policy. Real-world experiments on 8 everyday tasks show that our method can learn a diverse range of skills from a single human demonstration, whilst also yielding a stable and interpretable controller.

Key facts. Cited by 92 papers in the map (data through 7 Oct 2026). Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 32 of its 39 references that are in the map. First posted on arXiv 13 May 2021.

Builds on

32 of 39 references.

Closest ancestors · 4

Same direction, newest first

  1. 2020Visual Imitation Made Easy11 Aug
  2. 2019Self-Supervised Correspondence in Visuomotor Policy Learning16 Sep
  3. 2018Task-Embedded Control Networks for Few-Shot Imitation Learning8 Oct
  4. 2018Reinforcement and Imitation Learning for Diverse Visuomotor Skills26 Feb

Other references · 28

  1. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  2. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  3. 2016Generative Adversarial Imitation Learningcited by 1,576
  4. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  5. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 453
  6. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
  7. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
  8. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
  9. 2017Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperationcited by 286
  10. 2018Residual Reinforcement Learning for Robot Controlcited by 242
  11. 2017One-Shot Imitation Learningcited by 232
  12. 2017One-Shot Visual Imitation Learning via Meta-Learningcited by 225
16 more
  1. 2018One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learningcited by 146
  2. 2018Residual Policy Learningcited by 146
  3. 2019End-to-End Robotic Reinforcement Learning without Reward Engineeringcited by 115
  4. 2017Vision-Based Multi-Task Manipulation for Inexpensive Robots Using End-To-End Learning from Demonstrationcited by 100
  5. 2017Combining Self-Supervised Learning and Imitation for Vision-Based Rope Manipulationcited by 90
  6. 2017Transferring End-to-End Visuomotor Control from Simulation to Real World for a Multi-Stage Taskcited by 66
  7. 2019Deep Reinforcement Learning for Industrial Insertion Tasks with Visual Inputs and Natural Rewardscited by 58
  8. 2019Graph-Structured Visual Imitationcited by 43
  9. 2020KOVIS: Keypoint-based Visual Servoing with Zero-Shot Sim-to-Real Transfer for Robotics Manipulationcited by 16
  10. 2020FlowControl: Optical Flow Based Visual Servoingcited by 15
  11. 2019Generative predecessor models for sample-efficient imitation learningcited by 14
  12. 2020Intermittent Visual Servoing: Efficiently Learning Policies Robust to Instrument Changes for High-precision Surgical Manipulationcited by 13
  13. 2019Siamese Convolutional Neural Network for Sub-millimeter-accurate Camera Pose Estimation and Visual Servoingcited by 12
  14. 2021DROID: Minimizing the Reality Gap using Single-Shot Human Demonstrationcited by 11
  15. 2020SAFARI: Safe and Active Robot Imitation Learning with Imaginationcited by 5
  16. 2020Constrained-Space Optimization and Reinforcement Learning for Complex Taskscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 11; 2023: 24; 2024: 25; 2025: 15; 2026: 9.

92 papers.

Sort:

By area:

  1. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  2. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  3. 2024Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperationcited by 377
  4. 2023MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrationscited by 308
  5. 2023RVT: Robotic View Transformer for 3D Object Manipulationcited by 224
  6. 2024RT-H: Action Hierarchies Using Languagecited by 204
  7. 2022DALL-E-Bot: Introducing Web-Scale Diffusion Models to Roboticscited by 105
  8. 2022You Only Demonstrate Once: Category-Level Manipulation from Single Visual Demonstrationcited by 79
  9. 2022DexPoint: Generalizable Point Cloud Reinforcement Learning for Sim-to-Real Dexterous Manipulationcited by 78
  10. 2025BridgeVLA: Input-Output Alignment for Efficient 3D Manipulation Learning with Vision-Language Modelscited by 77
  11. 2023Teach a Robot to FISH: Versatile Imitation from One Minute of Demonstrationscited by 74
  12. 2022Can Foundation Models Perform Zero-Shot Task Specification For Robot Manipulation?cited by 70
38 more
  1. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learningcited by 68
  2. 2023DexArt: Benchmarking Generalizable Dexterous Manipulation with Articulated Objectscited by 64
  3. 2024OKAMI: Teaching Humanoid Robots Manipulation Skills through Single Video Imitationcited by 62
  4. 2023Waypoint-Based Imitation Learning for Robotic Manipulationcited by 59
  5. 2023Behavior Retrieval: Few-Shot Imitation Learning by Querying Unlabeled Datasetscited by 56
  6. 2023HYDRA: Hybrid Robot Actions for Imitation Learningcited by 54
  7. 2024SkillMimicGen: Automated Demonstration Generation for Efficient Skill Learning and Deploymentcited by 48
  8. 2024Vision-based Manipulation from Single Human Video with Open-World Object Graphscited by 46
  9. 2025VidBot: Learning Generalizable 3D Actions from In-the-Wild 2D Human Videos for Zero-Shot Robotic Manipulationcited by 46
  10. 2024DINOBot: Robot Manipulation via Retrieval and Alignment with Vision Foundation Modelscited by 43
  11. 2024Surgical Robot Transformer (SRT): Imitation Learning for Surgical Taskscited by 38
  12. 2022Demonstrate Once, Imitate Immediately (DOME): Learning Visual Servoing for One-Shot Imitation Learningcited by 36
  13. 2024R+X: Retrieval and Execution from Everyday Human Videoscited by 36
  14. 2023Prompt, Plan, Perform: LLM-based Humanoid Control via Quantized Imitation Learningcited by 31
  15. 2024IntervenGen: Interventional Data Generation for Robust and Data-Efficient Robot Imitation Learningcited by 24
  16. 2021Learning Multi-Stage Tasks with One Demonstration via Self-Replaycited by 21
  17. 2023One-Shot Imitation Learning: A Pose Estimation Perspectivecited by 21
  18. 2023On the Effectiveness of Retrieval, Alignment, and Replay in Manipulationcited by 15
  19. 2023Few-Shot In-Context Imitation Learning via Implicit Graph Alignmentcited by 14
  20. 2024Learning from Imperfect Demonstrations with Self-Supervision for Robotic Manipulationcited by 13
  21. 2025Responsive Noise-Relaying Diffusion Policy: Responsive and Efficient Visuomotor Controlcited by 13
  22. 2025Touch begins where vision ends: Generalizable policies for contact-rich manipulationcited by 13
  23. 2025FlowRAM: Grounding Flow Matching Policy with Region-Aware Mamba Framework for Robotic Manipulationcited by 13
  24. 2023A Survey of Demonstration Learningcited by 12
  25. 2024Diffusion Imitation from Observationcited by 12
  26. 2025A Survey on Imitation Learning for Contact-Rich Tasks in Roboticscited by 12
  27. 2024ILBiT: Imitation Learning for Robot Using Position and Torque Information based on Bilateral Control with Transformercited by 10
  28. 2024NaturalVLM: Leveraging Fine-grained Natural Language for Affordance-Guided Visual Manipulationcited by 10
  29. 2021Coarse-to-Fine for Sim-to-Real: Sub-Millimetre Precision Across Wide Task Spacescited by 9
  30. 2022Where To Start? Transferring Simple Skills to Complex Environmentscited by 9
  31. 2024MILES: Making Imitation Learning Easy with Self-Supervisioncited by 9
  32. 2021Learning Eye-in-Hand Camera Calibration from a Single Imagecited by 8
  33. 2023OpenVR: Teleoperation for Manipulationcited by 8
  34. 2023ForceSight: Text-Guided Mobile Manipulation with Visual-Force Goalscited by 8
  35. 2023NOD-TAMP: Generalizable Long-Horizon Planning with Neural Object Descriptorscited by 8
  36. 2024Adaptive Visual Imitation Learning for Robotic Assisted Feeding Across Varied Bowl Configurations and Food Typescited by 8
  37. 2025Spatial-Temporal Aware Visuomotor Diffusion Policy Learningcited by 8
  38. 2025One-Shot Dual-Arm Imitation Learningcited by 7

50 most cited of 92 All 92 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Human-to-Robot Interaction: Learning from Video Demonstration for Robot Imitation
  2. 2025Robotic Manipulation via Imitation Learning: Taxonomy, Evolution, Benchmark, and Challenges
  3. 2021The Surprising Effectiveness of Representation Learning for Visual Imitation
  4. 2023MimicPlay: Long-Horizon Imitation Learning by Watching Human Play
  5. 2021Towards More Generalizable One-shot Visual Imitation Learning
  6. 2021Manipulator-Independent Representations for Visual Imitation
  7. 2020Learning to Generalize Across Long-Horizon Tasks from Human Demonstrations
  8. 2022Dexterous Imitation Made Easy: A Learning-Based Framework for Efficient Dexterous Manipulation