mlsanityData to 7 Oct 2026

3D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representations

arXiv 2403.03954 ↗First posted 6 Mar 2024cs.RO cs.CV cs.LGcited by 115

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

Imitation learning provides an efficient way to teach robots dexterous skills; however, learning complex skills robustly and generalizablely usually consumes large amounts of human demonstrations. To tackle this challenging problem, we present 3D Diffusion Policy (DP3), a novel visual imitation learning approach that incorporates the power of 3D visual representations into diffusion policies, a class of conditional action generative models. The core design of DP3 is the utilization of a compact 3D visual representation, extracted from sparse point clouds with an efficient point encoder. In our experiments involving 72 simulation tasks, DP3 successfully handles most tasks with just 10 demonstrations and surpasses baselines with a 24.2% relative improvement. In 4 real robot tasks, DP3 demonstrates precise control with a high success rate of 85%, given only 40 demonstrations of each task, and shows excellent generalization abilities in diverse aspects, including space, viewpoint, appearance, and instance. Interestingly, in real robot experiments, DP3 rarely violates safety requirements, in contrast to baseline methods which frequently do, necessitating human intervention. Our extensive evaluation highlights the critical importance of 3D representations in real-world robot learning. Videos, code, and data are available on https://3d-diffusion-policy.github.io .

Key facts. Cited by 115 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 79 of its 88 references that are in the map. First posted on arXiv 6 Mar 2024.

Builds on

79 of 88 references.

Closest ancestors · 10

Same direction, newest first

  1. 20243D Diffuser Actor: Policy Diffusion with 3D Scene Representations16 Feb
  2. 2023Diffusion Reward: Learning Rewards via Conditional Video Diffusion21 Dec
  3. 2023EDMP: Ensemble-of-costs-guided Diffusion for Motion Planning20 Sep
  4. 2023Extracting Reward Functions from Diffusion Models1 Jun
  5. 2023Goal-Conditioned Imitation Learning using Score-based Diffusion Policies5 Apr
  6. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusion7 Mar
  7. 2023Imitating Human Behaviour with Diffusion Models25 Jan
  8. 2022Is Conditional Generative Modeling all you need for Decision-Making?28 Nov
  9. 2022SE(3)-DiffusionFields: Learning smooth cost functions for joint grasp and motion optimization through diffusion8 Sep
  10. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 69

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2020Denoising Diffusion Implicit Modelscited by 9,655
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  6. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  7. 2016Layer Normalizationcited by 5,579
  8. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  9. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  10. 2020Point Transformercited by 1,462
  11. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  12. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
38 more
  1. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Modelscited by 924
  2. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 882
  3. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  4. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  5. 2020SAPIEN: A SimulAted Part-based Interactive ENvironmentcited by 584
  6. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  7. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 542
  8. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 484
  9. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 482
  10. 2021Implicit Behavioral Cloningcited by 442
  11. 2022PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategiescited by 432
  12. 2024Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperationcited by 377
  13. 2024Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robotscited by 373
  14. 2022Behavior Transformers: Cloning k modes with one stonecited by 299
  15. 2020Learning Agile Robotic Locomotion Skills by Imitating Animalscited by 229
  16. 2023RVT: Robotic View Transformer for 3D Object Manipulationcited by 224
  17. 2023MimicPlay: Long-Horizon Imitation Learning by Watching Human Playcited by 214
  18. 2021DexMV: Imitation Learning for Dexterous Manipulation from Human Videoscited by 194
  19. 2022Masked World Models for Visual Controlcited by 169
  20. 2023Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisitioncited by 156
  21. 2021A System for General In-Hand Object Re-Orientationcited by 155
  22. 2021The Surprising Effectiveness of Representation Learning for Visual Imitationcited by 149
  23. 2022DeXtreme: Transfer of Agile In-hand Manipulation from Simulation to Realitycited by 136
  24. 2023AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation Systemcited by 136
  25. 2023Distilled Feature Fields Enable Few-Shot Language-Guided Manipulationcited by 130
  26. 2022Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learningcited by 113
  27. 2019DexPilot: Vision Based Teleoperation of Dexterous Robotic Hand-Arm Systemcited by 105
  28. 2022In-Hand Object Rotation via Rapid Motor Adaptationcited by 102
  29. 2021Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentationcited by 101
  30. 2023On Bringing Robots Homecited by 101
  31. 2021PlasticineLab: A Soft-Body Manipulation Benchmark with Differentiable Physicscited by 100
  32. 2023GNFactor: Multi-Task Real Robot Learning with Generalizable Neural Feature Fieldscited by 99
  33. 2022Visual Dexterity: In-Hand Reorientation of Novel and Complex Object Shapescited by 92
  34. 2023General In-Hand Object Rotation with Vision and Touchcited by 87
  35. 2023Rotating without Seeing: Towards In-hand Dexterity through Touchcited by 78
  36. 2022Dexterous Imitation Made Easy: A Learning-Based Framework for Efficient Dexterous Manipulationcited by 75
  37. 2023RoboCook: Long-Horizon Elasto-Plastic Object Manipulation with Diverse Toolscited by 75
  38. 2023Teach a Robot to FISH: Versatile Imitation from One Minute of Demonstrationscited by 74

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 40; 2025: 55; 2026: 20.

115 papers.

Sort:

By area:

  1. 2025RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulationcited by 477
  2. 2024DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Drivingcited by 324
  3. 2025DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledgecited by 203
  4. 2025HybridVLA: Collaborative Diffusion and Autoregression in a Unified Vision-Language-Action Modelcited by 176
  5. 2024Consistency Policy: Accelerated Visuomotor Policies via Consistency Distillationcited by 159
  6. 2024DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulationcited by 149
  7. 2025RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twinscited by 117
  8. 2026StarVLA: A Lego-like Codebase for Vision-Language-Action Model Developingcited by 103
  9. 2024One-Step Diffusion Policy: Fast Visuomotor Policies via Diffusion Distillationcited by 83
  10. 2024Towards Synergistic, Generalized, and Efficient Dual-System for Robotic Manipulationcited by 71
  11. 2024RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twins (early version)cited by 68
  12. 2025WorldEval: World Model as Real-World Robot Policies Evaluatorcited by 64
38 more
  1. 2024OKAMI: Teaching Humanoid Robots Manipulation Skills through Single Video Imitationcited by 62
  2. 2025Motion Tracks: A Unified Representation for Human-Robot Transfer in Few-Shot Imitation Learningcited by 57
  3. 2024Sparse Diffusion Policy: A Sparse, Reusable, and Flexible Policy for Robot Learningcited by 56
  4. 2024GenSim2: Scaling Robot Data Generation with Multi-modal and Reasoning LLMscited by 48
  5. 2025ManiFlow: A General Robot Manipulation Policy via Consistency Flow Trainingcited by 43
  6. 2024G3Flow: Generative 3D Semantic Flow for Pose-aware and Generalizable Object Manipulationcited by 35
  7. 2024ARCap: Collecting High-quality Human Demonstrations for Robot Learning with Augmented Reality Feedbackcited by 31
  8. 2025DRAWER: Digital Reconstruction and Articulation With Environment Realismcited by 29
  9. 2024SPA: 3D Spatial-Awareness Enables Effective Embodied Representationcited by 28
  10. 2025Learning Long-Context Diffusion Policies via Past-Token Predictioncited by 28
  11. 2025Fast-in-Slow: A Dual-System Foundation Model Unifying Fast Manipulation within Slow Reasoningcited by 28
  12. 2024M2Diffuser: Diffusion-based Trajectory Optimization for Mobile Manipulation in 3D Scenescited by 26
  13. 2024ActionFlow: Equivariant, Accurate, and Efficient Policies with Spatially Symmetric Flow Matchingcited by 25
  14. 2024Grounding Video Models to Actions through Goal Conditioned Explorationcited by 25
  15. 2025Spatial-Temporal Graph Diffusion Policy with Kinematic Modeling for Bimanual Robotic Manipulationcited by 24
  16. 2024DNAct: Diffusion Guided Multi-Task 3D Policy Learningcited by 22
  17. 2024Lessons from Learning to Spin "Pens"cited by 22
  18. 2024Mamba Policy: Towards Efficient 3D Diffusion Policy with Hybrid Selective State Modelscited by 22
  19. 2025MLA: A Multisensory Language-Action Model for Multimodal Understanding and Forecasting in Robotic Manipulationcited by 22
  20. 2025SKIL: Semantic Keypoint Imitation Learning for Generalizable Data-efficient Manipulationcited by 21
  21. 2025Being-0: A Humanoid Robotic Agent with Vision-Language Models and Modular Skillscited by 21
  22. 2025AC-DiT: Adaptive Coordination Diffusion Transformer for Mobile Manipulationcited by 21
  23. 2024A Self-Correcting Vision-Language-Action Model for Fast and Slow System Manipulationcited by 20
  24. 2024Real-World Offline Reinforcement Learning from Vision Language Model Feedbackcited by 20
  25. 2026HoloBrain-0 Technical Reportcited by 20
  26. 2024GarmentLab: A Unified Simulation and Benchmark for Garment Manipulationcited by 19
  27. 2026Global Prior Meets Local Consistency: Dual-Memory Augmented Vision-Language-Action Model for Efficient Robotic Manipulationcited by 19
  28. 2024Imagination Policy: Using Generative Point Cloud Models for Learning Manipulation Policiescited by 18
  29. 2024Generative Image as Action Modelscited by 18
  30. 2024Diffusion Policy Attacker: Crafting Adversarial Attacks for Diffusion-based Policiescited by 17
  31. 2024DiffPhyCon: A Generative Approach to Control Complex Physical Systemscited by 17
  32. 2025Curating Demonstrations using Online Experiencecited by 16
  33. 2025Adapt3R: Adaptive 3D Scene Representation for Domain Transfer in Imitation Learningcited by 16
  34. 2024Learning Visual Quadrupedal Loco-Manipulation from Demonstrationscited by 15
  35. 2024Continuous Control with Coarse-to-fine Reinforcement Learningcited by 15
  36. 2025FDPP: Fine-tune Diffusion Policy with Human Preferencecited by 14
  37. 2025Falcon: Fast Visuomotor Policies via Partial Denoisingcited by 14
  38. 2024Leveraging Locality to Boost Sample Efficiency in Robotic Manipulationcited by 13

50 most cited of 115 All 115 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026R3D: Revisiting 3D Policy Learning
  2. 2025Spatial-Temporal Aware Visuomotor Diffusion Policy Learning
  3. 20253D Flow Diffusion Policy: Visuomotor Policy Learning via Generating Flow in 3D Space
  4. 2025ISS Policy : Scalable Diffusion Policy with Implicit Scene Supervision
  5. 2025VO-DP: Semantic-Geometric Adaptive Diffusion Policy for Vision-Only Robotic Manipulation
  6. 20254D Visual Pre-training for Robot Learning
  7. 2026Spatially Conditioned Diffusion Policy: Learning Precise and Robust Manipulation with a Single RGB Camera
  8. 2025H3DP: Triply-Hierarchical Diffusion Policy for Visuomotor Learning