mlsanityData to 7 Oct 2026

3D Diffuser Actor: Policy Diffusion with 3D Scene Representations

arXiv 2402.10885 ↗First posted 16 Feb 2024cs.RO cs.AI cs.CVcited by 278

On the map: Diffusion models → Diffusion and flow policies for robotics

Abstract (arXiv)

Diffusion policies are conditional diffusion models that learn robot action distributions conditioned on the robot and environment state. They have recently shown to outperform both deterministic and alternative action distribution learning formulations. 3D robot policies use 3D scene feature representations aggregated from a single or multiple camera views using sensed depth. They have shown to generalize better than their 2D counterparts across camera viewpoints. We unify these two lines of work and present 3D Diffuser Actor, a neural policy equipped with a novel 3D denoising transformer that fuses information from the 3D visual scene, a language instruction and proprioception to predict the noise in noised 3D robot pose trajectories. 3D Diffuser Actor sets a new state-of-the-art on RLBench with an absolute performance gain of 18.1% over the current SOTA on a multi-view setup and an absolute gain of 13.1% on a single-view setup. On the CALVIN benchmark, it improves over the current SOTA by a 9% relative increase. It also learns to control a robot manipulator in the real world from a handful of demonstrations. Through thorough comparisons with the current SOTA policies and ablations of our model, we show 3D Diffuser Actor's design choices dramatically outperform 2D representations, regression and classification objectives, absolute attentions, and holistic non-tokenized 3D scene embeddings.

Key facts. Cited by 278 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 74 of its 126 references that are in the map. First posted on arXiv 16 Feb 2024.

Builds on

74 of 126 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024Diffusion-ES: Gradient-free Planning with Diffusion for Autonomous Driving and Zero-Shot Instruction Following9 Feb
  2. 2023Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Models16 Oct
  3. 2023Revisiting Energy Based Models as Policies: Ranking Noise Contrastive Estimation and Interpolating Energy Models11 Sep
  4. 2023Diffusion-EDFs: Bi-equivariant Denoising Generative Modeling on SE(3) for Visual Robotic Manipulation6 Sep
  5. 2023DiMSam: Diffusion Models as Samplers for Task and Motion Planning under Partial Observability22 Jun
  6. 2023Diffusion Model is an Effective Planner and Data Synthesizer for Multi-Task Reinforcement Learning29 May
  7. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies20 Apr
  8. 2023Goal-Conditioned Imitation Learning using Score-based Diffusion Policies5 Apr
  9. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusion7 Mar
  10. 2023ReorientDiff: Diffusion Model based Reorientation for Object Manipulation28 Feb
  11. 2023AdaptDiffuser: Diffusion Models as Adaptive Self-evolving Planners3 Feb
  12. 2023Imitating Human Behaviour with Diffusion Models25 Jan
3 more
  1. 2022DALL-E-Bot: Introducing Web-Scale Diffusion Models to Robotics5 Oct
  2. 2022SE(3)-DiffusionFields: Learning smooth cost functions for joint grasp and motion optimization through diffusion8 Sep
  3. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 59

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2020Denoising Diffusion Implicit Modelscited by 9,655
  4. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  5. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  8. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  9. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  10. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  11. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  12. 2016Generative Adversarial Imitation Learningcited by 1,576
38 more
  1. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,186
  2. 2018Self-Attention with Relative Position Representationscited by 1,112
  3. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  4. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  5. 2016End to End Learning for Self-Driving Carscited by 1,000
  6. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  7. 2022A Generalist Agentcited by 802
  8. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
  9. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  10. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
  11. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  12. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  13. 2023Learning Universal Policies via Text-Guided Video Generationcited by 506
  14. 2021Implicit Behavioral Cloningcited by 442
  15. 2022PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategiescited by 432
  16. 2023Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulationcited by 353
  17. 2023Vision-Language Foundation Models as Effective Robot Imitatorscited by 344
  18. 2020Transporter Networks: Rearranging the Visual World for Robotic Manipulationcited by 342
  19. 2022Behavior Transformers: Cloning k modes with one stonecited by 299
  20. 2023RVT: Robotic View Transformer for 3D Object Manipulationcited by 224
  21. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 208
  22. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 177
  23. 2019Goal-conditioned Imitation Learningcited by 159
  24. 2022What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Datacited by 137
  25. 2023Act3D: 3D Feature Field Transformers for Multi-Task Robotic Manipulationcited by 135
  26. 2021Coarse-to-Fine Q-attention: Efficient Learning for Visual Robotic Manipulation via Discretisationcited by 120
  27. 2022From Play to Policy: Conditional Behavior Generation from Uncurated Robot Datacited by 103
  28. 2023GenAug: Retargeting behaviors to unseen situations via Generative Augmentationcited by 103
  29. 2022Grounding Language with Visual Affordances over Unstructured Datacited by 101
  30. 2023GNFactor: Multi-Task Real Robot Learning with Generalizable Neural Feature Fieldscited by 99
  31. 2022Instruction-driven history-aware policies for robotic manipulationscited by 93
  32. 2023Compositional Foundation Models for Hierarchical Planningcited by 92
  33. 2014Reducing the Barrier to Entry of Complex Robotic Software: a MoveIt! Case Studycited by 86
  34. 2019IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Datacited by 85
  35. 2021Q-attention: Enabling Efficient Learning for Vision-based Robotic Manipulationcited by 85
  36. 2021Lepard: Learning partial point cloud matching in rigid and deformable scenescited by 85
  37. 2020Learning to Rearrange Deformable Cables, Fabrics, and Bags with Goal-Conditioned Transporter Networkscited by 77
  38. 2022CACTI: A Framework for Scalable Multi-Task Multi-Scene Visual Imitation Learningcited by 70

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 53; 2025: 122; 2026: 103.

278 papers.

Sort:

By area:

  1. 2025RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulationcited by 477
  2. 2024GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulationcited by 290
  3. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  4. 2025DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledgecited by 203
  5. 2025DexVLA: Vision-Language Model with Plug-In Diffusion Expert for General Robot Controlcited by 191
  6. 2025HybridVLA: Collaborative Diffusion and Autoregression in a Unified Vision-Language-Action Modelcited by 176
  7. 2024RoboMIND: Benchmark on Multi-embodiment Intelligence Normative Data for Robot Manipulationcited by 166
  8. 2024Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulationcited by 159
  9. 2025HAMSTER: Hierarchical Action Models For Open-World Robot Manipulationcited by 125
  10. 20243D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representationscited by 115
  11. 2025PointVLA: Injecting the 3D World into Vision-Language-Action Modelscited by 101
  12. 2024Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimizationcited by 99
38 more
  1. 2025UP-VLA: A Unified Understanding and Prediction Model for Embodied Agentcited by 99
  2. 2024What Matters in Building Vision-Language-Action Models for Generalist Robotscited by 88
  3. 2024One-Step Diffusion Policy: Fast Visuomotor Policies via Diffusion Distillationcited by 83
  4. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  5. 2025OpenHelix: A Short Survey, Empirical Analysis, and Open-Source Dual-System VLA Model for Robotic Manipulationcited by 79
  6. 2025BridgeVLA: Input-Output Alignment for Efficient 3D Manipulation Learning with Vision-Language Modelscited by 77
  7. 2025Dita: Scaling Diffusion Transformer for Generalist Vision-Language-Action Policycited by 73
  8. 2024The Ingredients for Robotic Diffusion Transformerscited by 72
  9. 2024Towards Synergistic, Generalized, and Efficient Dual-System for Robotic Manipulationcited by 71
  10. 2024Moto: Latent Motion Token as the Bridging Language for Learning Robot Manipulation from Videoscited by 65
  11. 2025WorldEval: World Model as Real-World Robot Policies Evaluatorcited by 64
  12. 2025Efficient Online Reinforcement Learning for Diffusion Policycited by 61
  13. 2026Being-H0.5: Scaling Human-Centric Robot Learning for Cross-Embodiment Generalizationcited by 61
  14. 2025EnerVerse: Envisioning Embodied Future Space for Robotics Manipulationcited by 59
  15. 2024ManiCM: Real-time 3D Diffusion Policy via Consistency Model for Robotic Manipulationcited by 57
  16. 2024Diffusion-VLA: Generalizable and Interpretable Robot Foundation Model via Self-Generated Reasoningcited by 57
  17. 2024Sparse Diffusion Policy: A Sparse, Reusable, and Flexible Policy for Robot Learningcited by 56
  18. 2025FLOWER: Democratizing Generalist Robot Policies with Efficient Vision-Language-Action Flow Policiescited by 56
  19. 2024Efficient Diffusion Transformer Policies with Mixture of Expert Denoisers for Multitask Learningcited by 46
  20. 2025NavDP: Learning Sim-to-Real Navigation Diffusion Policy with Privileged Information Guidancecited by 46
  21. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 46
  22. 2024VidMan: Exploiting Implicit Dynamics from Video Diffusion Model for Effective Robot Manipulationcited by 45
  23. 2024PhysMotion: Physics-Grounded Dynamics From a Single Imagecited by 45
  24. 2024Lift3D Foundation Policy: Lifting 2D Large-Scale Pretrained Models for Robust 3D Robotic Manipulationcited by 45
  25. 2025World-in-World: World Models in a Closed-Loop Worldcited by 44
  26. 2025ManiFlow: A General Robot Manipulation Policy via Consistency Flow Trainingcited by 43
  27. 2024From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessonscited by 42
  28. 2026Xiaomi-Robotics-0: An Open-Sourced Vision-Language-Action Model with Real-Time Executioncited by 42
  29. 2024Diffusion Transformer Policycited by 41
  30. 2024GR-MG: Leveraging Partially Annotated Data via Multi-Modal Goal-Conditioned Policycited by 39
  31. 2025Diffusion Models for Robotic Manipulation: A Surveycited by 39
  32. 2024Towards Generalizable Vision-Language Robotic Manipulation: A Benchmark and LLM-guided 3D Policycited by 37
  33. 2024Grounding Multimodal Large Language Models in Actionscited by 35
  34. 2024Deep Generative Models in Robotics: A Survey on Learning from Multimodal Demonstrationscited by 35
  35. 2024View-Invariant Policy Learning via Zero-Shot Novel View Synthesiscited by 35
  36. 2024G3Flow: Generative 3D Semantic Flow for Pose-aware and Generalizable Object Manipulationcited by 35
  37. 2025Much Ado About Noising: Dispelling the Myths of Generative Robotic Controlcited by 35
  38. 2024From LLMs to Actions: Latent Codes as Bridges in Hierarchical Robot Controlcited by 34

50 most cited of 278 All 278 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Prediction with Action: Visual Policy Learning via Joint Denoising Process
  2. 2025Exploration and Adaptation in Non-Stationary Tasks with Diffusion Policies
  3. 2024Unpacking the Individual Components of Diffusion Policy
  4. 2026DiffusionAnything: End-to-End In-context Diffusion Learning for Unified Navigation and Pre-Grasp Motion
  5. 2025Conditioning Matters: Training Diffusion Policies is Faster Than You Think
  6. 2026Immiscible Diffusion Policy: Preserving Multimodal Robot Actions through Label-Free Noise Assignment
  7. 2026Beyond Future Prediction: Denoising as Generative Adaptation for Robot Control
  8. 2025Pixel Motion Diffusion is What We Need for Robot Control