mlsanityData to 7 Oct 2026

One-shot Visual Imitation via Attributed Waypoints and Demonstration Augmentation

arXiv 2302.04856 ↗First posted 9 Feb 2023cs.RO cs.CV cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

In this paper, we analyze the behavior of existing techniques and design new solutions for the problem of one-shot visual imitation. In this setting, an agent must solve a novel instance of a novel task given just a single visual demonstration. Our analysis reveals that current methods fall short because of three errors: the DAgger problem arising from purely offline training, last centimeter errors in interacting with objects, and mis-fitting to the task context rather than to the actual task. This motivates the design of our modular approach where we a) separate out task inference (what to do) from task execution (how to do it), and b) develop data augmentation and generation techniques to mitigate mis-fitting. The former allows us to leverage hand-crafted motor primitives for task execution which side-steps the DAgger problem and last centimeter errors, while the latter gets the model to focus on the task rather than the task context. Our model gets 100% and 48% success rates on two recent benchmarks, improving upon the current state-of-the-art by absolute 90% and 20% respectively.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 45 of its 57 references that are in the map. First posted on arXiv 9 Feb 2023.

Builds on

45 of 57 references.

References · 45

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  4. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  5. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  6. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  7. 2016Generative Adversarial Imitation Learningcited by 1,576
  8. 2019MixMatch: A Holistic Approach to Semi-Supervised Learningcited by 1,390
  9. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  10. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  11. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  12. 2020Reinforcement Learning with Augmented Datacited by 452
33 more
  1. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 380
  2. 2016Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collectioncited by 312
  3. 2020Learning to Explore using Active Neural SLAMcited by 308
  4. 2016Deep Visual Foresight for Planning Robot Motioncited by 276
  5. 2017Soft-DTW: a Differentiable Loss Function for Time-Seriescited by 265
  6. 2018Visual Reinforcement Learning with Imagined Goalscited by 248
  7. 2017One-Shot Imitation Learningcited by 232
  8. 2017One-Shot Visual Imitation Learning via Meta-Learningcited by 225
  9. 2015Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hourscited by 208
  10. 2022Human-to-Robot Imitation in the Wildcited by 157
  11. 2018One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learningcited by 146
  12. 2018Zero-Shot Visual Imitationcited by 114
  13. 2019AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videoscited by 105
  14. 2021XIRL: Cross-embodiment Inverse Reinforcement Learningcited by 101
  15. 2018Driving Policy Transfer via Modularity and Abstractioncited by 91
  16. 2021Learning by Watching: Physical Imitation of Manipulation Skills from Human Videoscited by 87
  17. 2020Transformers for One-Shot Visual Imitationcited by 77
  18. 2021Augmenting Reinforcement Learning with Behavior Primitives for Diverse Manipulation Taskscited by 75
  19. 2019Third-Person Visual Imitation Learning via Decoupled Hierarchical Controllercited by 73
  20. 2018Task-Embedded Control Networks for Few-Shot Imitation Learningcited by 65
  21. 2021Accelerating Robotic Reinforcement Learning via Parameterized Action Primitivescited by 63
  22. 2020Learning Robot Skills with Temporal Variational Inferencecited by 54
  23. 2019Combining Optimal Control and Learning for Visual Navigation in Novel Environmentscited by 50
  24. 2021Towards More Generalizable One-shot Visual Imitation Learningcited by 48
  25. 2019Efficient Bimanual Manipulation Using Learned Task Schemascited by 44
  26. 2019Learning One-Shot Imitation from Humans without Humanscited by 32
  27. 2018Deep Drone Racing: Learning Agile Flight in Dynamic Environmentscited by 31
  28. 2018Beauty and the Beast: Optimal Methods Meet Learning for Drone Racingcited by 20
  29. 2020Modeling Long-horizon Tasks as Sequential Interaction Landscapescited by 14
  30. 2020A Geometric Perspective on Visual Imitation Learningcited by 10
  31. 2019Continuous Relaxation of Symbolic Planner for One-Shot Imitation Learningcited by 8
  32. 2019Learning Navigation Subroutines from Egocentric Videoscited by 7
  33. 2019Neural Autonomous Navigation with Riemannian Motion Policycited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 1.

4 papers.

  1. 2024OKAMI: Teaching Humanoid Robots Manipulation Skills through Single Video Imitationcited by 62
  2. 2025World Models for Learning Dexterous Hand-Object Interactions from Human Videoscited by 26
  3. 2025See Once, Then Act: Vision-Language-Action Model with Task Learning from One-Shot Video Demonstrationscited by 4
  4. 2026Unifying Object-Centric World Models and Diffusion Policy: A Hierarchical Framework for Multi-Stage Robotic Taskscited by 2