# POIL: Point-based One-Shot Imitation Learning with Stable Dynamical Systems

arXiv: https://arxiv.org/abs/2609.30404 · HTML: https://mlsanity.com/paper/2609.30404/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 32 of its 41 references that are in the map.
- First posted on arXiv 24 Sep 2026.

## Placed in

- Offline RL and imitation learning from data / [Robot manipulation imitation learning](https://mlsanity.com/offlinerl/robot-manipulation-imitation-learning/)

## Builds on (most cited first, 30 of 32)

- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Open X-Embodiment: Robotic Learning Datasets and RT-X Models](https://mlsanity.com/paper/2310.08864/) (2023)
- [RLBench: The Robot Learning Benchmark & Learning Environment](https://mlsanity.com/paper/1909.12271/) (2019)
- [CoTracker: It is Better to Track Together](https://mlsanity.com/paper/2307.07635/) (2023)
- [Emergent Correspondence from Image Diffusion](https://mlsanity.com/paper/2306.03881/) (2023)
- [ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation](https://mlsanity.com/paper/2409.01652/) (2024)
- [One-Shot Imitation Learning](https://mlsanity.com/paper/1703.07326/) (2017)
- [One-Shot Visual Imitation Learning via Meta-Learning](https://mlsanity.com/paper/1709.04905/) (2017)
- [kPAM: KeyPoint Affordances for Category-Level Robotic Manipulation](https://mlsanity.com/paper/1903.06684/) (2019)
- [MOKA: Open-World Robotic Manipulation through Mark-Based Visual Prompting](https://mlsanity.com/paper/2403.03174/) (2024)
- [CoTracker3: Simpler and Better Point Tracking by Pseudo-Labelling Real Videos](https://mlsanity.com/paper/2410.11831/) (2024)
- [RoboTAP: Tracking Arbitrary Points for Few-Shot Visual Imitation](https://mlsanity.com/paper/2308.15975/) (2023)
- [Telling Left from Right: Identifying Geometry-Aware Semantic Correspondence](https://mlsanity.com/paper/2311.17034/) (2023)
- [Detect Anything via Next Point Prediction](https://mlsanity.com/paper/2510.12798/) (2025)
- [KETO: Learning Keypoint Representations for Tool Manipulation](https://mlsanity.com/paper/1910.11977/) (2019)
- [Dynamic Movement Primitives in Robotics: A Tutorial Survey](https://mlsanity.com/paper/2102.03861/) (2021)
- [Point Policy: Unifying Observations and Actions with Key Points for Robot Manipulation](https://mlsanity.com/paper/2502.20391/) (2025)
- [DINOBot: Robot Manipulation via Retrieval and Alignment with Vision Foundation Models](https://mlsanity.com/paper/2402.13181/) (2024)
- [Instant Policy: In-Context Imitation Learning via Graph Diffusion](https://mlsanity.com/paper/2411.12633/) (2024)
- [kPAM 2.0: Feedback Control for Category-Level Robotic Manipulation](https://mlsanity.com/paper/2102.06279/) (2021)
- [Demonstrate Once, Imitate Immediately (DOME): Learning Visual Servoing for One-Shot Imitation Learning](https://mlsanity.com/paper/2204.02863/) (2022)
- [D3Fields: Dynamic 3D Descriptor Fields for Zero-Shot Generalizable Rearrangement](https://mlsanity.com/paper/2309.16118/) (2023)
- [K-VIL: Keypoints-based Visual Imitation Learning](https://mlsanity.com/paper/2209.03277/) (2022)
- [Multi-View 3D Point Tracking](https://mlsanity.com/paper/2508.21060/) (2025)
- [MimicFunc: Imitating Tool Manipulation from a Single Human Video via Functional Correspondence](https://mlsanity.com/paper/2508.13534/) (2025)
- [One-Shot Manipulation Strategy Learning by Making Contact Analogies](https://mlsanity.com/paper/2411.09627/) (2024)
- [Elastic Motion Policy: An Adaptive Dynamical System for Robust and Efficient One-Shot Imitation Learning](https://mlsanity.com/paper/2503.08029/) (2025)

## Built on by (most cited first, 1 of 1)

- [In-Context Learning for Robots: Methods and Applications](https://mlsanity.com/paper/2609.36012/) (2026)

## Abstract (arXiv)

We present POIL, a point-based one-shot imitation learning framework with stable dynamical systems. While one-shot imitation avoids collecting extensive demonstrations, successful one-shot manipulation requires not only transferring a demonstrated trajectory to a novel object but also executing it robustly under changing scene conditions, grasp configurations, and external disturbances. POIL addresses both problems through a shared representation: a set of 3D points on the object's functional part, used jointly for trajectory transfer and closed-loop execution. The one-shot transfer from the demonstrated trajectory is enabled with point correspondences. POIL grounds the shared functional part with a multi-modal large language model, and transfers the trajectory across viewpoint, pose, and object category changes. During execution, multi-view tracking observes the same points online, and Point-set BCSDM drives them in closed loop by projecting per-point velocities onto a single rigid-body twist computed from the tracked points alone. This extends stable dynamical models from an SE(3) pose to a point set without requiring a known 3D model or pose estimator. We show that at the goal the controller becomes a gradient flow on the classical SO(3) potential, so its terminal phase inherits the almost-global convergence of that potential under a rigid-object assumption. Across simulation and real-robot experiments, POIL transfers a single demonstration across object category, grasp pose, and goal geometry, while recovering from external disturbances during execution. Project page: https://sangminkim-99.github.io/poil