# Graph-Structured Visual Imitation

arXiv: https://arxiv.org/abs/1907.05518 · HTML: https://mlsanity.com/paper/1907.05518/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Jul 2019.

## Built on by (most cited first, 30 of 43)

- [VIMA: General Robot Manipulation with Multimodal Prompts](https://mlsanity.com/paper/2210.03094/) (2022)
- [MimicPlay: Long-Horizon Imitation Learning by Watching Human Play](https://mlsanity.com/paper/2302.12422/) (2023)
- [DexMV: Imitation Learning for Dexterous Manipulation from Human Videos](https://mlsanity.com/paper/2108.05877/) (2021)
- [HumanPlus: Humanoid Shadowing and Imitation from Humans](https://mlsanity.com/paper/2406.10454/) (2024)
- [Act3D: 3D Feature Field Transformers for Multi-Task Robotic Manipulation](https://mlsanity.com/paper/2306.17817/) (2023)
- [VIOLA: Imitation Learning for Vision-Based Manipulation with Object Proposal Priors](https://mlsanity.com/paper/2210.11339/) (2022)
- [Coarse-to-Fine Imitation Learning: Robot Manipulation from a Single Demonstration](https://mlsanity.com/paper/2105.06411/) (2021)
- [Learning by Watching: Physical Imitation of Manipulation Skills from Human Videos](https://mlsanity.com/paper/2101.07241/) (2021)
- [From One Hand to Multiple Hands: Imitation Learning for Dexterous Manipulation from Single-Camera Teleoperation](https://mlsanity.com/paper/2204.12490/) (2022)
- [You Only Demonstrate Once: Category-Level Manipulation from Single Visual Demonstration](https://mlsanity.com/paper/2201.12716/) (2022)
- [XSkill: Cross Embodiment Skill Discovery](https://mlsanity.com/paper/2307.09955/) (2023)
- [Gen2Sim: Scaling up Robot Learning in Simulation with Generative Models](https://mlsanity.com/paper/2310.18308/) (2023)
- [Learning Generalizable Manipulation Policies with Object-Centric 3D Representations](https://mlsanity.com/paper/2310.14386/) (2023)
- [Robot See Robot Do: Imitating Articulated Object Manipulation with Monocular 4D Reconstruction](https://mlsanity.com/paper/2409.18121/) (2024)
- [Zero-Shot Robot Manipulation from Passive Human Videos](https://mlsanity.com/paper/2302.02011/) (2023)
- [Graph Inverse Reinforcement Learning from Diverse Videos](https://mlsanity.com/paper/2207.14299/) (2022)
- [FluidLab: A Differentiable Environment for Benchmarking Complex Fluid Manipulation](https://mlsanity.com/paper/2303.02346/) (2023)
- [Towards Generalist Robot Learning from Internet Video: A Survey](https://mlsanity.com/paper/2404.19664/) (2024)
- [Deep Generative Models in Robotics: A Survey on Learning from Multimodal Demonstrations](https://mlsanity.com/paper/2408.04380/) (2024)
- [Energy-based Models are Zero-Shot Planners for Compositional Scene Rearrangement](https://mlsanity.com/paper/2304.14391/) (2023)
- [K-VIL: Keypoints-based Visual Imitation Learning](https://mlsanity.com/paper/2209.03277/) (2022)
- [Efficient and Interpretable Robot Manipulation with Graph Neural Networks](https://mlsanity.com/paper/2102.13177/) (2021)
- [One-Shot Imitation Learning: A Pose Estimation Perspective](https://mlsanity.com/paper/2310.12077/) (2023)
- [Composing Pre-Trained Object-Centric Representations for Robotics From "What" and "Where" Foundation Models](https://mlsanity.com/paper/2404.13474/) (2024)
- [P3-PO: Prescriptive Point Priors for Visuo-Spatial Generalization of Robot Policies](https://mlsanity.com/paper/2412.06784/) (2024)
- [Few-Shot In-Context Imitation Learning via Implicit Graph Alignment](https://mlsanity.com/paper/2310.12238/) (2023)
- [One-Shot Imitation under Mismatched Execution](https://mlsanity.com/paper/2409.06615/) (2024)
- [Learning by Watching: A Review of Video-based Learning Approaches for Robot Manipulation](https://mlsanity.com/paper/2402.07127/) (2024)
- [A Geometric Perspective on Visual Imitation Learning](https://mlsanity.com/paper/2003.02768/) (2020)
- [VIEW: Visual Imitation Learning with Waypoints](https://mlsanity.com/paper/2404.17906/) (2024)

## Abstract (arXiv)

We cast visual imitation as a visual correspondence problem. Our robotic agent is rewarded when its actions result in better matching of relative spatial configurations for corresponding visual entities detected in its workspace and teacher's demonstration. We build upon recent advances in Computer Vision,such as human finger keypoint detectors, object detectors trained on-the-fly with synthetic augmentations, and point detectors supervised by viewpoint changes and learn multiple visual entity detectors for each demonstration without human annotations or robot interactions. We empirically show the proposed factorized visual representations of entities and their spatial arrangements drive successful imitation of a variety of manipulation skills within minutes, using a single demonstration and without any environment instrumentation. It is robust to background clutter and can effectively generalize across environment variations between demonstrator and imitator, greatly outperforming unstructured non-factorized full-frame CNN encodings of previous works.