mlsanityData to 7 Oct 2026

Imagination Policy: Using Generative Point Cloud Models for Learning Manipulation Policies

arXiv 2406.11740 ↗First posted 17 Jun 2024cs.RO cs.AI cs.LGcited by 18

Not on the map. Why?

Abstract (arXiv)

Humans can imagine goal states during planning and perform actions to match those goals. In this work, we propose Imagination Policy, a novel multi-task key-frame policy network for solving high-precision pick and place tasks. Instead of learning actions directly, Imagination Policy generates point clouds to imagine desired states which are then translated to actions using rigid action estimation. This transforms action inference into a local generative task. We leverage pick and place symmetries underlying the tasks in the generation process and achieve extremely high sample efficiency and generalizability to unseen configurations. Finally, we demonstrate state-of-the-art performance across various tasks on the RLbench benchmark compared with several strong baselines and validate our approach on a real robot.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 49 of its 56 references that are in the map. First posted on arXiv 17 Jun 2024.

Builds on

49 of 56 references.

References · 49

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023Segment Anythingcited by 8,158
  3. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  4. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  5. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  6. 2023Consistency Modelscited by 1,778
  7. 2019Normalizing Flows for Probabilistic Modeling and Inferencecited by 1,315
  8. 2023Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inferencecited by 826
  9. 2021Diffusion Probabilistic Models for 3D Point Cloud Generationcited by 734
  10. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
  11. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  12. 2022LION: Latent Point Diffusion Models for 3D Shape Generationcited by 552
37 more
  1. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  2. 20213D Shape Generation and Completion through Point-Voxel Diffusioncited by 516
  3. 2019PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flowscited by 487
  4. 2019General E(2)-Equivariant Steerable CNNscited by 441
  5. 2023InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generationcited by 387
  6. 2023Segment Anything in High Qualitycited by 373
  7. 2021Knowledge Distillation in Iterative Generative Models for Improved Sampling Speedcited by 344
  8. 2019Point-Voxel CNN for Efficient 3D Deep Learningcited by 343
  9. 2021Vector Neurons: A General Framework for SO(3)-Equivariant Networkscited by 304
  10. 20243D Diffuser Actor: Policy Diffusion with 3D Scene Representationscited by 278
  11. 2022Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphscited by 258
  12. 2023RVT: Robotic View Transformer for 3D Object Manipulationcited by 224
  13. 2021Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulationcited by 204
  14. 20193D Point Cloud Generative Adversarial Network Based on Tree Structured Graph Convolutionscited by 173
  15. 2022Fast Sampling of Diffusion Models via Operator Learningcited by 173
  16. 2016Generative and Discriminative Voxel Modeling with Convolutional Neural Networkscited by 128
  17. 20243D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representationscited by 115
  18. 2021Q-attention: Enabling Efficient Learning for Vision-based Robotic Manipulationcited by 85
  19. 2022SO(2)-Equivariant Reinforcement Learningcited by 79
  20. 2021SetVAE: Learning Hierarchical Composition for Generative Modeling of Set-Structured Datacited by 78
  21. 2021Equivariant Q Learning in Spatial Action Spacescited by 61
  22. 2022Fast Point Cloud Generation with Straight Flowscited by 56
  23. 2022Equivariant Descriptor Fields: SE(3)-Equivariant Energy-Based Models for End-to-End Visual Robotic Manipulation Learningcited by 54
  24. 2022Sample Efficient Grasp Learning Using Equivariant Modelscited by 52
  25. 2022SE(3)-Equivariant Relational Rearrangement with Neural Descriptor Fieldscited by 51
  26. 2023Diffusion-EDFs: Bi-equivariant Denoising Generative Modeling on SE(3) for Visual Robotic Manipulationcited by 51
  27. 2022TAX-Pose: Task-Specific Cross-Pose Estimation for Robot Manipulationcited by 47
  28. 2022Equivariant Transporter Networkcited by 44
  29. 2023KITE: Keypoint-Conditioned Policies for Semantic Manipulationcited by 39
  30. 2022On-Robot Learning With Equivariant Modelscited by 38
  31. 2023EquivAct: SIM(3)-Equivariant Visuomotor Policies beyond Rigid Object Manipulationcited by 37
  32. 2022Edge Grasp Network: A Graph-Based SE(3)-invariant Approach to Grasp Detectioncited by 34
  33. 2022The Surprising Effectiveness of Equivariant Models in Domains with Latent Symmetrycited by 32
  34. 2024Fourier Transporter: Bi-Equivariant Robotic Manipulation in 3Dcited by 32
  35. 2023Shelving, Stacking, Hanging: Relational Pose Diffusion for Multi-modal Rearrangementcited by 29
  36. 2024Deep SE(3)-Equivariant Geometric Reasoning for Precise Placement Taskscited by 19
  37. 2023Leveraging Symmetries in Pick and Placecited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 12; 2026: 4.

18 papers.

Sort:

By area:

  1. 2025NovaFlow: Zero-Shot Manipulation via Actionable Flow from Generated Videoscited by 38
  2. 2025Rectified Point Flow: Generic Point Cloud Pose Estimationcited by 13
  3. 2025Two by Two: Learning Multi-Task Pairwise Objects Assembly for Generalizable Robot Manipulationcited by 12
  4. 2025GP3: A 3D Geometry-Aware Policy with Multi-View Images for Robotic Manipulationcited by 11
  5. 2025AnyPlace: Learning Generalized Object Placement for Robot Manipulationcited by 10
  6. 2025SE(3)-Equivariant Diffusion Policy in Spherical Fourier Spacecited by 7
  7. 2026NovaPlan: Zero-Shot Long-Horizon Manipulation via Closed-Loop Video Language Planningcited by 7
  8. 2024MATCH POLICY: A Simple Pipeline from Point Cloud Registration to Manipulation Policiescited by 6
  9. 2024Loss Distillation via Gradient Matching for Point Cloud Completion with Weighted Chamfer Distancecited by 4
  10. 2025A Practical Guide for Incorporating Symmetry in Diffusion Policycited by 4
  11. 2025Coarse-to-Fine 3D Keyframe Transportercited by 2
  12. 2025Generalizable Hierarchical Skill Learning via Object-Centric Representationcited by 2
6 more
  1. 2025Equivariant Volumetric Graspingcited by 1
  2. 2025Multi-Group Equivariant Augmentation for Reinforcement Learning in Robot Manipulationcited by 0
  3. 2025DynaRend: Learning 3D Dynamics via Masked Future Rendering for Robotic Manipulationcited by 0
  4. 2026Action Map Policy: Learning 3D Closed-loop Manipulation via Pixel Classificationcited by 0
  5. 2026Pix2Act: Image-Space Manipulation Policies with Equivariant Augmentationcited by 0
  6. 2026World Motion Models: Flexible Sequence Modeling of SE(3) Trajectoriescited by 0