# One-shot Imitation Learning via Interaction Warping

arXiv: https://arxiv.org/abs/2306.12392 · HTML: https://mlsanity.com/paper/2306.12392/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 33 of its 47 references that are in the map.
- First posted on arXiv 21 Jun 2023.

## Placed in

- Offline RL and imitation learning from data / [Robot manipulation imitation learning](https://mlsanity.com/offlinerl/robot-manipulation-imitation-learning/)

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Dynamic Graph CNN for Learning on Point Clouds](https://mlsanity.com/paper/1801.07829/) (2018)
- [Detecting Twenty-thousand Classes using Image-level Supervision](https://mlsanity.com/paper/2201.02605/) (2022)
- [Simple Open-Vocabulary Object Detection with Vision Transformers](https://mlsanity.com/paper/2205.06230/) (2022)
- [Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation](https://mlsanity.com/paper/2112.05124/) (2021)
- [kPAM: KeyPoint Affordances for Category-Level Robotic Manipulation](https://mlsanity.com/paper/1903.06684/) (2019)
- [Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulation](https://mlsanity.com/paper/1806.08756/) (2018)
- [A comprehensive survey on point cloud registration](https://mlsanity.com/paper/2103.02690/) (2021)
- [SO(2)-Equivariant Reinforcement Learning](https://mlsanity.com/paper/2203.04439/) (2022)
- [You Only Demonstrate Once: Category-Level Manipulation from Single Visual Demonstration](https://mlsanity.com/paper/2201.12716/) (2022)
- [Keypoints into the Future: Self-Supervised Correspondence in Model-Based Reinforcement Learning](https://mlsanity.com/paper/2009.05085/) (2020)
- [KETO: Learning Keypoint Representations for Tool Manipulation](https://mlsanity.com/paper/1910.11977/) (2019)
- [Explorations in Homeomorphic Variational Auto-Encoding](https://mlsanity.com/paper/1807.04689/) (2018)
- [Equivariant Descriptor Fields: SE(3)-Equivariant Energy-Based Models for End-to-End Visual Robotic Manipulation Learning](https://mlsanity.com/paper/2206.08321/) (2022)
- [Learning Symmetric Embeddings for Equivariant World Models](https://mlsanity.com/paper/2204.11371/) (2022)
- [SE(3)-Equivariant Relational Rearrangement with Neural Descriptor Fields](https://mlsanity.com/paper/2211.09786/) (2022)
- [TAX-Pose: Task-Specific Cross-Pose Estimation for Robot Manipulation](https://mlsanity.com/paper/2211.09325/) (2022)
- [kPAM 2.0: Feedback Control for Category-Level Robotic Manipulation](https://mlsanity.com/paper/2102.06279/) (2021)
- [S3K: Self-Supervised Semantic Keypoints for Robotic Manipulation via Multi-View Consistency](https://mlsanity.com/paper/2009.14711/) (2020)
- [kPAM-SC: Generalizable Manipulation Planning using KeyPoint Affordance and Shape Completion](https://mlsanity.com/paper/1909.06980/) (2019)
- [Local Neural Descriptor Fields: Locally Conditioned Object Representations for Manipulation](https://mlsanity.com/paper/2302.03573/) (2023)
- [GIFT: Generalizable Interaction-aware Functional Tool Affordances without Labels](https://mlsanity.com/paper/2106.14973/) (2021)
- [Transferring Category-based Functional Grasping Skills by Latent Space Non-Rigid Registration](https://mlsanity.com/paper/1809.05390/) (2018)
- [Transferring Grasping Skills to Novel Instances by Latent Space Non-Rigid Registration](https://mlsanity.com/paper/1809.05353/) (2018)
- [Supervised Autonomous Locomotion and Manipulation for Disaster Response with a Centaur-like Robot](https://mlsanity.com/paper/1809.06802/) (2018)

## Built on by (most cited first, 16 of 16)

- [Sparse Diffusion Policy: A Sparse, Reusable, and Flexible Policy for Robot Learning](https://mlsanity.com/paper/2407.01531/) (2024)
- [Diffusion-EDFs: Bi-equivariant Denoising Generative Modeling on SE(3) for Visual Robotic Manipulation](https://mlsanity.com/paper/2309.02685/) (2023)
- [Keypoint Abstraction using Large Models for Object-Relative Imitation Learning](https://mlsanity.com/paper/2410.23254/) (2024)
- [One-Shot Imitation Learning with Invariance Matching for Robotic Manipulation](https://mlsanity.com/paper/2405.13178/) (2024)
- [MimicFunc: Imitating Tool Manipulation from a Single Human Video via Functional Correspondence](https://mlsanity.com/paper/2508.13534/) (2025)
- [FUNCTO: Function-Centric One-Shot Imitation Learning for Tool Manipulation](https://mlsanity.com/paper/2502.11744/) (2025)
- [One-Shot Manipulation Strategy Learning by Making Contact Analogies](https://mlsanity.com/paper/2411.09627/) (2024)
- [One-Shot Dual-Arm Imitation Learning](https://mlsanity.com/paper/2503.06831/) (2025)
- [Elastic Motion Policy: An Adaptive Dynamical System for Robust and Efficient One-Shot Imitation Learning](https://mlsanity.com/paper/2503.08029/) (2025)
- [Visual Imitation Learning of Task-Oriented Object Grasping and Rearrangement](https://mlsanity.com/paper/2403.14000/) (2024)
- [One-shot Video Imitation via Parameterized Symbolic Abstraction Graphs](https://mlsanity.com/paper/2408.12674/) (2024)
- [RAPID: Robot Agentic Programming from Demonstrations](https://mlsanity.com/paper/2609.30249/) (2026)
- [Annotation-Free One-Shot Imitation Learning for Multi-Step Manipulation Tasks](https://mlsanity.com/paper/2509.24972/) (2025)
- [Multi-Group Equivariant Augmentation for Reinforcement Learning in Robot Manipulation](https://mlsanity.com/paper/2508.11204/) (2025)
- [RA-VLA: Retrieval-Augmented VLA for Test-Time Adaptation](https://mlsanity.com/paper/2608.25585/) (2026)
- [MemCorr-DP: Counterfactual Correspondence Conditioning for a Diffusion Policy Guided by a Reference](https://mlsanity.com/paper/2609.06615/) (2026)

## Abstract (arXiv)

Imitation learning of robot policies from few demonstrations is crucial in open-ended applications. We propose a new method, Interaction Warping, for learning SE(3) robotic manipulation policies from a single demonstration. We infer the 3D mesh of each object in the environment using shape warping, a technique for aligning point clouds across object instances. Then, we represent manipulation actions as keypoints on objects, which can be warped with the shape of the object. We show successful one-shot imitation learning on three simulated and real-world object re-arrangement tasks. We also demonstrate the ability of our method to predict object meshes and robot grasps in the wild.