# Instant Policy: In-Context Imitation Learning via Graph Diffusion

arXiv: https://arxiv.org/abs/2411.12633 · HTML: https://mlsanity.com/paper/2411.12633/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 40 of its 48 references that are in the map.
- First posted on arXiv 19 Nov 2024.

## Placed in

- Offline RL and imitation learning from data / [Robot manipulation imitation learning](https://mlsanity.com/offlinerl/robot-manipulation-imitation-learning/)

## Builds on (most cited first, 30 of 40)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [RLBench: The Robot Learning Benchmark & Learning Environment](https://mlsanity.com/paper/1909.12271/) (2019)
- [Perceiver IO: A General Architecture for Structured Inputs & Outputs](https://mlsanity.com/paper/2107.14795/) (2021)
- [MediaPipe: A Framework for Building Perception Pipelines](https://mlsanity.com/paper/1906.08172/) (2019)
- [Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation](https://mlsanity.com/paper/2209.05451/) (2022)
- [BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning](https://mlsanity.com/paper/2202.02005/) (2022)
- [Masked Label Prediction: Unified Message Passing Model for Semi-Supervised Classification](https://mlsanity.com/paper/2009.03509/) (2020)
- [OGB-LSC: A Large-Scale Challenge for Machine Learning on Graphs](https://mlsanity.com/paper/2103.09430/) (2021)
- [MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations](https://mlsanity.com/paper/2310.17596/) (2023)
- [Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Models](https://mlsanity.com/paper/2310.10639/) (2023)
- [One-Shot Imitation Learning](https://mlsanity.com/paper/1703.07326/) (2017)
- [Scaling Robot Learning with Semantically Imagined Experience](https://mlsanity.com/paper/2302.11550/) (2023)
- [ALOHA Unleashed: A Simple Recipe for Robot Dexterity](https://mlsanity.com/paper/2410.13126/) (2024)
- [Heterogeneous Information Network Embedding for Recommendation](https://mlsanity.com/paper/1711.10730/) (2017)
- [SE(3)-DiffusionFields: Learning smooth cost functions for joint grasp and motion optimization through diffusion](https://mlsanity.com/paper/2209.03855/) (2022)
- [DALL-E-Bot: Introducing Web-Scale Diffusion Models to Robotics](https://mlsanity.com/paper/2210.02438/) (2022)
- [Keypoint Action Tokens Enable In-Context Imitation Learning in Robotics](https://mlsanity.com/paper/2403.19578/) (2024)
- [In-Context Imitation Learning via Next-Token Prediction](https://mlsanity.com/paper/2408.15980/) (2024)

## Built on by (most cited first, 30 of 39)

- [Exploring the Limits of Vision-Language-Action Manipulations in Cross-task Generalization](https://mlsanity.com/paper/2505.15660/) (2025)
- [MAP-VLA: Memory-Augmented Prompting for Vision-Language-Action Model in Robotic Manipulation](https://mlsanity.com/paper/2511.09516/) (2025)
- [SKIL: Semantic Keypoint Imitation Learning for Generalizable Data-efficient Manipulation](https://mlsanity.com/paper/2501.14400/) (2025)
- [AirExo-2: Scaling up Generalizable Robotic Imitation Learning with Low-Cost Exoskeletons](https://mlsanity.com/paper/2503.03081/) (2025)
- [Action Tokenizer Matters in In-Context Imitation Learning](https://mlsanity.com/paper/2503.01206/) (2025)
- [Act-Observe-Rewrite: Multimodal Coding Agents as In-Context Policy Learners for Robot Manipulation](https://mlsanity.com/paper/2603.04466/) (2026)
- [One-Shot Dual-Arm Imitation Learning](https://mlsanity.com/paper/2503.06831/) (2025)
- [In-Context Robot Learning with VLM Agents](https://mlsanity.com/paper/2609.19138/) (2026)
- [Learning Affordances at Inference-Time for Vision-Language-Action Models](https://mlsanity.com/paper/2510.19752/) (2025)
- [X-Diffusion: Training Diffusion Policies on Cross-Embodiment Human Demonstrations](https://mlsanity.com/paper/2511.04671/) (2025)
- [IA-VLA: Input Augmentation for Vision-Language-Action models in settings with semantically complex tasks](https://mlsanity.com/paper/2509.24768/) (2025)
- [OMP: One-step Meanflow Policy with Directional Alignment](https://mlsanity.com/paper/2512.19347/) (2025)
- [MaNGO - Adaptable Graph Network Simulators via Meta-Learning](https://mlsanity.com/paper/2510.05874/) (2025)
- [Knowledge-Driven Imitation Learning: Enabling Generalization Across Diverse Conditions](https://mlsanity.com/paper/2506.21057/) (2025)
- [Learning in ImaginationLand: Omnidirectional Policies through 3D Generative Models (OP-Gen)](https://mlsanity.com/paper/2509.06191/) (2025)
- [Generalizable Insights for Graph Transformers in Theory and Practice](https://mlsanity.com/paper/2511.08028/) (2025)
- [Ada3Drift: Adaptive Training-Time Drifting for One-Step 3D Visuomotor Robotic Manipulation](https://mlsanity.com/paper/2603.11984/) (2026)
- [Instant-Fold: In-Context Imitation Learning for Deformable Object Manipulation](https://mlsanity.com/paper/2606.04269/) (2026)
- [Large Pre-Trained Models for Bimanual Manipulation in 3D](https://mlsanity.com/paper/2509.20579/) (2025)
- [Annotation-Free One-Shot Imitation Learning for Multi-Step Manipulation Tasks](https://mlsanity.com/paper/2509.24972/) (2025)
- [RADAR: Closed-Loop Robotic Data Generation via Semantic Planning and Autonomous Causal Environment Reset](https://mlsanity.com/paper/2603.11811/) (2026)
- [A Hierarchical Spatiotemporal Action Tokenizer for In-Context Imitation Learning in Robotics](https://mlsanity.com/paper/2604.15215/) (2026)
- [SynthICL: Scalable In-context Imitation Learning with Synthetic Data](https://mlsanity.com/paper/2606.08154/) (2026)
- [Difference-Aware Retrieval Policies for Imitation Learning](https://mlsanity.com/paper/2606.09758/) (2026)
- [Adaptation of Generalist Robot Policies with Minimal Data](https://mlsanity.com/paper/2608.11363/) (2026)
- [ContextFlow: In-Context Flow Matching for Robot Manipulation](https://mlsanity.com/paper/2609.06852/) (2026)
- [Generalizing Manipulation Skills with a Local Coding Agent](https://mlsanity.com/paper/2609.26499/) (2026)
- [POIL: Point-based One-Shot Imitation Learning with Stable Dynamical Systems](https://mlsanity.com/paper/2609.30404/) (2026)
- [In-Context Learning for Robots: Methods and Applications](https://mlsanity.com/paper/2609.36012/) (2026)
- [ManiLong-Shot: Interaction-Aware One-Shot Imitation Learning for Long-Horizon Manipulation](https://mlsanity.com/paper/2512.16302/) (2025)

## Abstract (arXiv)

Following the impressive capabilities of in-context learning with large transformers, In-Context Imitation Learning (ICIL) is a promising opportunity for robotics. We introduce Instant Policy, which learns new tasks instantly (without further training) from just one or two demonstrations, achieving ICIL through two key components. First, we introduce inductive biases through a graph representation and model ICIL as a graph generation problem with a learned diffusion process, enabling structured reasoning over demonstrations, observations, and actions. Second, we show that such a model can be trained using pseudo-demonstrations - arbitrary trajectories generated in simulation - as a virtually infinite pool of training data. Simulated and real experiments show that Instant Policy enables rapid learning of various everyday robot tasks. We also show how it can serve as a foundation for cross-embodiment and zero-shot transfer to language-defined tasks. Code and videos are available at https://www.robot-learning.uk/instant-policy.