# Imagination Policy: Using Generative Point Cloud Models for Learning Manipulation Policies

arXiv: https://arxiv.org/abs/2406.11740 · HTML: https://mlsanity.com/paper/2406.11740/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 56 references that are in the map.
- First posted on arXiv 17 Jun 2024.

## Builds on (most cited first, 30 of 49)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [Consistency Models](https://mlsanity.com/paper/2303.01469/) (2023)
- [Normalizing Flows for Probabilistic Modeling and Inference](https://mlsanity.com/paper/1912.02762/) (2019)
- [Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference](https://mlsanity.com/paper/2310.04378/) (2023)
- [Diffusion Probabilistic Models for 3D Point Cloud Generation](https://mlsanity.com/paper/2103.01458/) (2021)
- [RLBench: The Robot Learning Benchmark & Learning Environment](https://mlsanity.com/paper/1909.12271/) (2019)
- [Perceiver IO: A General Architecture for Structured Inputs & Outputs](https://mlsanity.com/paper/2107.14795/) (2021)
- [LION: Latent Point Diffusion Models for 3D Shape Generation](https://mlsanity.com/paper/2210.06978/) (2022)
- [Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation](https://mlsanity.com/paper/2209.05451/) (2022)
- [3D Shape Generation and Completion through Point-Voxel Diffusion](https://mlsanity.com/paper/2104.03670/) (2021)
- [PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flows](https://mlsanity.com/paper/1906.12320/) (2019)
- [General E(2)-Equivariant Steerable CNNs](https://mlsanity.com/paper/1911.08251/) (2019)
- [InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation](https://mlsanity.com/paper/2309.06380/) (2023)
- [Segment Anything in High Quality](https://mlsanity.com/paper/2306.01567/) (2023)
- [Knowledge Distillation in Iterative Generative Models for Improved Sampling Speed](https://mlsanity.com/paper/2101.02388/) (2021)
- [Point-Voxel CNN for Efficient 3D Deep Learning](https://mlsanity.com/paper/1907.03739/) (2019)
- [Vector Neurons: A General Framework for SO(3)-Equivariant Networks](https://mlsanity.com/paper/2104.12229/) (2021)
- [3D Diffuser Actor: Policy Diffusion with 3D Scene Representations](https://mlsanity.com/paper/2402.10885/) (2024)
- [Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphs](https://mlsanity.com/paper/2206.11990/) (2022)
- [RVT: Robotic View Transformer for 3D Object Manipulation](https://mlsanity.com/paper/2306.14896/) (2023)
- [Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation](https://mlsanity.com/paper/2112.05124/) (2021)
- [Fast Sampling of Diffusion Models via Operator Learning](https://mlsanity.com/paper/2211.13449/) (2022)
- [3D Point Cloud Generative Adversarial Network Based on Tree Structured Graph Convolutions](https://mlsanity.com/paper/1905.06292/) (2019)
- [Generative and Discriminative Voxel Modeling with Convolutional Neural Networks](https://mlsanity.com/paper/1608.04236/) (2016)
- [3D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representations](https://mlsanity.com/paper/2403.03954/) (2024)
- [Q-attention: Enabling Efficient Learning for Vision-based Robotic Manipulation](https://mlsanity.com/paper/2105.14829/) (2021)

## Built on by (most cited first, 18 of 18)

- [NovaFlow: Zero-Shot Manipulation via Actionable Flow from Generated Videos](https://mlsanity.com/paper/2510.08568/) (2025)
- [Rectified Point Flow: Generic Point Cloud Pose Estimation](https://mlsanity.com/paper/2506.05282/) (2025)
- [Two by Two: Learning Multi-Task Pairwise Objects Assembly for Generalizable Robot Manipulation](https://mlsanity.com/paper/2504.06961/) (2025)
- [GP3: A 3D Geometry-Aware Policy with Multi-View Images for Robotic Manipulation](https://mlsanity.com/paper/2509.15733/) (2025)
- [AnyPlace: Learning Generalized Object Placement for Robot Manipulation](https://mlsanity.com/paper/2502.04531/) (2025)
- [SE(3)-Equivariant Diffusion Policy in Spherical Fourier Space](https://mlsanity.com/paper/2507.01723/) (2025)
- [NovaPlan: Zero-Shot Long-Horizon Manipulation via Closed-Loop Video Language Planning](https://mlsanity.com/paper/2602.20119/) (2026)
- [MATCH POLICY: A Simple Pipeline from Point Cloud Registration to Manipulation Policies](https://mlsanity.com/paper/2409.15517/) (2024)
- [A Practical Guide for Incorporating Symmetry in Diffusion Policy](https://mlsanity.com/paper/2505.13431/) (2025)
- [Loss Distillation via Gradient Matching for Point Cloud Completion with Weighted Chamfer Distance](https://mlsanity.com/paper/2409.06171/) (2024)
- [Coarse-to-Fine 3D Keyframe Transporter](https://mlsanity.com/paper/2502.01773/) (2025)
- [Generalizable Hierarchical Skill Learning via Object-Centric Representation](https://mlsanity.com/paper/2510.21121/) (2025)
- [Equivariant Volumetric Grasping](https://mlsanity.com/paper/2507.18847/) (2025)
- [DynaRend: Learning 3D Dynamics via Masked Future Rendering for Robotic Manipulation](https://mlsanity.com/paper/2510.24261/) (2025)
- [Multi-Group Equivariant Augmentation for Reinforcement Learning in Robot Manipulation](https://mlsanity.com/paper/2508.11204/) (2025)
- [Action Map Policy: Learning 3D Closed-loop Manipulation via Pixel Classification](https://mlsanity.com/paper/2607.10706/) (2026)
- [Pix2Act: Image-Space Manipulation Policies with Equivariant Augmentation](https://mlsanity.com/paper/2607.11167/) (2026)
- [World Motion Models: Flexible Sequence Modeling of SE(3) Trajectories](https://mlsanity.com/paper/2610.01742/) (2026)

## Abstract (arXiv)

Humans can imagine goal states during planning and perform actions to match those goals. In this work, we propose Imagination Policy, a novel multi-task key-frame policy network for solving high-precision pick and place tasks. Instead of learning actions directly, Imagination Policy generates point clouds to imagine desired states which are then translated to actions using rigid action estimation. This transforms action inference into a local generative task. We leverage pick and place symmetries underlying the tasks in the generation process and achieve extremely high sample efficiency and generalizability to unseen configurations. Finally, we demonstrate state-of-the-art performance across various tasks on the RLbench benchmark compared with several strong baselines and validate our approach on a real robot.