mlsanityData to 14 Oct 2026

SE(3)-DiffusionFields: Learning smooth cost functions for joint grasp and motion optimization through diffusion

arXiv 2209.03855 ↗First posted 8 Sep 2022cs.RO cs.LGcited by 127

On the map: Diffusion models → Diffusion and flow policies for robotics

Abstract (arXiv)

Multi-objective optimization problems are ubiquitous in robotics, e.g., the optimization of a robot manipulation task requires a joint consideration of grasp pose configurations, collisions and joint limits. While some demands can be easily hand-designed, e.g., the smoothness of a trajectory, several task-specific objectives need to be learned from data. This work introduces a method for learning data-driven SE(3) cost functions as diffusion models. Diffusion models can represent highly-expressive multimodal distributions and exhibit proper gradients over the entire space due to their score-matching training objective. Learning costs as diffusion models allows their seamless integration with other costs into a single differentiable objective function, enabling joint gradient-based motion optimization. In this work, we focus on learning SE(3) diffusion models for 6DoF grasping, giving rise to a novel framework for joint grasp and motion optimization without needing to decouple grasp selection from trajectory generation. We evaluate the representation power of our SE(3) diffusion models w.r.t. classical generative models, and we showcase the superior performance of our proposed optimization framework in a series of simulated and real-world robotic manipulation tasks against representative baselines.

Key facts. Cited by 127 papers in the map (data through 14 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 49 of its 79 references that are in the map. First posted on arXiv 8 Sep 2022.

Builds on

49 of 79 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 48

  1. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  2. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  3. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,897
  4. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,689
  5. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,636
  6. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  7. 2020Improved Techniques for Training Score-Based Generative Modelscited by 1,043
  8. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 998
  9. 2017PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenescited by 775
  10. 2021Neural Fields in Visual Computing and Beyondcited by 540
  11. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 482
  12. 2017Towards Principled Methods for Training Generative Adversarial Networkscited by 464
36 more
  1. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 406
  2. 2022Understanding Diffusion Models: A Unified Perspectivecited by 340
  3. 2021Vector Neurons: A General Framework for SO(3)-Equivariant Networkscited by 305
  4. 2021How to Train Your Energy-Based Modelscited by 274
  5. 2017Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metricscited by 264
  6. 20196-DOF GraspNet: Variational Grasp Generation for Object Manipulationcited by 242
  7. 2022Riemannian Score-Based Generative Modellingcited by 219
  8. 2021Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenescited by 218
  9. 2021Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulationcited by 209
  10. 2018A micro Lie theory for state estimation in roboticscited by 124
  11. 2022Riemannian Diffusion Modelscited by 122
  12. 2020ACRONYM: A Large-Scale Grasp Dataset Based on Simulationcited by 117
  13. 2017Grasp Pose Detection in Point Cloudscited by 111
  14. 2022iSDF: Real-Time Neural Signed Distance Fields for Robot Perceptioncited by 109
  15. 2021Synergies Between Affordance and Geometry: 6-DoF Grasp Detection via Implicit Representationscited by 108
  16. 2020se(3)-TrackNet: Data-driven 6D Pose Tracking by Calibrating Image Residuals in Synthetic Domainscited by 90
  17. 2018PointNetGPD: Detecting Grasp Configurations from Point Setscited by 80
  18. 20196-DOF Grasping for Target-driven Object Manipulation in Cluttercited by 70
  19. 2022Theseus: A Library for Differentiable Nonlinear Optimizationcited by 65
  20. 2021STORM: An Integrated Framework for Fast Joint-Space Model-Predictive Control for Reactive Manipulationcited by 47
  21. 2020Object Rearrangement Using Learned Implicit Collision Functionscited by 43
  22. 2018Deep Energy Estimator Networkscited by 34
  23. 2020ImitationFlow: Learning Deep Stable Stochastic Dynamic Systems by Normalizing Flowscited by 33
  24. 2021Learning Models as Functionals of Signed-Distance Fields for Manipulation Planningcited by 29
  25. 2019Discriminator optimal transportcited by 25
  26. 2020Encoding Physical Constraints in Differentiable Newton-Euler Algorithmcited by 23
  27. 2022Learning Implicit Priors for Motion Optimizationcited by 20
  28. 2018Planning Multi-Fingered Grasps as Probabilistic Inference in a Learned Deep Networkcited by 19
  29. 2019Manipulation Trajectory Optimization with Online Grasp Synthesis and Selectioncited by 17
  30. 2021Composable Energy Policies for Reactive Motion Generation and Reinforcement Learningcited by 16
  31. 2022Regularized Deep Signed Distance Fields for Reactive Motion Generationcited by 16
  32. 2021Benchmarking Structured Policies and Policy Optimization for Real-World Dexterous Object Manipulationcited by 15
  33. 2021Motion Planning by Learning the Solution Manifold in Trajectory Optimizationcited by 13
  34. 2021Collision-Aware Target-Driven Object Grasping in Constrained Environmentscited by 11
  35. 2022Score-Based Generative Models for Molecule Generationcited by 10
  36. 2021GraspME -- Grasp Manifold Estimatorcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 23; 2024: 48; 2025: 39; 2026: 15.

127 papers.

Sort:

By area:

  1. 20243D Diffuser Actor: Policy Diffusion with 3D Scene Representationscited by 282
  2. 2023Goal-Conditioned Imitation Learning using Score-based Diffusion Policiescited by 236
  3. 2025HybridVLA: Collaborative Diffusion and Autoregression in a Unified Vision-Language-Action Modelcited by 181
  4. 2023Motion Planning Diffusion: Learning and Planning of Robot Motions with Diffusion Modelscited by 161
  5. 2023Distilled Feature Fields Enable Few-Shot Language-Guided Manipulationcited by 130
  6. 20243D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representationscited by 116
  7. 2024One-Step Diffusion Policy: Fast Visuomotor Policies via Diffusion Distillationcited by 89
  8. 2024Prediction with Action: Visual Policy Learning via Joint Denoising Processcited by 70
  9. 2023Enhancing Deep Reinforcement Learning: A Tutorial on Generative Diffusion Models in Network Optimizationcited by 56
  10. 2024Streaming Diffusion Policy: Fast Policy Synthesis with Variable Noise Diffusion Modelscited by 56
  11. 2023Diffusion-EDFs: Bi-equivariant Denoising Generative Modeling on SE(3) for Visual Robotic Manipulationcited by 52
  12. 2024Learning Multimodal Behaviors from Scratch with Diffusion Policy Gradientcited by 51
38 more
  1. 2022StructDiffusion: Language-Guided Creation of Physically-Valid Structures using Unseen Objectscited by 50
  2. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 49
  3. 2024DexDiffuser: Generating Dexterous Grasps with Diffusion Modelscited by 48
  4. 2024Efficient Diffusion Transformer Policies with Mixture of Expert Denoisers for Multitask Learningcited by 48
  5. 2024PoCo: Policy Composition from and for Heterogeneous Robot Learningcited by 42
  6. 2024Potential Based Diffusion Motion Planningcited by 41
  7. 2025Diffusion Models for Robotic Manipulation: A Surveycited by 41
  8. 2024Instant Policy: In-Context Imitation Learning via Graph Diffusioncited by 39
  9. 2024Scaling Diffusion Policy in Transformer to 1 Billion Parameters for Robotic Manipulationcited by 38
  10. 2024Inference-Time Policy Steering through Human Interactionscited by 38
  11. 2024A Survey of Embodied Learning for Object-Centric Robotic Manipulationcited by 35
  12. 2023SE(3) Diffusion Model-based Point Cloud Registration for Robust 6D Object Pose Estimationcited by 33
  13. 2024Motion Planning Diffusion: Learning and Adapting Robot Motion Planning with Diffusion Modelscited by 33
  14. 2025FP3: A 3D Foundation Policy for Robotic Manipulationcited by 32
  15. 2025GraspGen: A Diffusion-based Framework for 6-DOF Grasping with On-Generator Trainingcited by 32
  16. 2023Shelving, Stacking, Hanging: Relational Pose Diffusion for Multi-modal Rearrangementcited by 29
  17. 2023Refining Diffusion Planner for Reliable Behavior Synthesis by Automatic Detection of Infeasible Planscited by 27
  18. 2024M2Diffuser: Diffusion-based Trajectory Optimization for Mobile Manipulation in 3D Scenescited by 27
  19. 2024Language-driven Grasp Detectioncited by 26
  20. 2024ActionFlow: Equivariant, Accurate, and Efficient Policies with Spatially Symmetric Flow Matchingcited by 26
  21. 2024ET-SEED: Efficient Trajectory-Level SE(3) Equivariant Diffusion Policycited by 26
  22. 2024GenDP: 3D Semantic Fields for Category-Level Generalizable Diffusion Policycited by 25
  23. 2025Prior Does Matter: Visual Navigation via Denoising Diffusion Bridge Modelscited by 25
  24. 20236D-Diff: A Keypoint Diffusion Framework for 6D Object Pose Estimationcited by 24
  25. 2024DNAct: Diffusion Guided Multi-Task 3D Policy Learningcited by 23
  26. 2023DiMSam: Diffusion Models as Samplers for Task and Motion Planning under Partial Observabilitycited by 22
  27. 2023SparseDFF: Sparse-View Feature Distillation for One-Shot Dexterous Manipulationcited by 22
  28. 2023Generative Modeling on Manifolds Through Mixture of Riemannian Diffusion Processescited by 21
  29. 2024Don't Start from Scratch: Behavioral Refinement via Interpolant-based Policy Diffusioncited by 21
  30. 2024Neural Fields in Robotics: A Surveycited by 21
  31. 2024Discrete Policy: Learning Disentangled Action Space for Multi-Task Robotic Manipulationcited by 19
  32. 2023Grasp Multiple Objects with One Handcited by 18
  33. 2024Generative Image as Action Modelscited by 18
  34. 2023Confronting Ambiguity in 6D Object Pose Estimation via Score-Based Diffusion on SE(3)cited by 17
  35. 2024Composable Part-Based Manipulationcited by 17
  36. 2023Dream2Real: Zero-Shot 3D Object Rearrangement with Vision-Language Modelscited by 16
  37. 2024ICGNet: A Unified Approach for Instance-Centric Graspingcited by 16
  38. 2024Language-Driven 6-DoF Grasp Detection Using Negative Prompt Guidancecited by 16

50 most cited of 127 All 127 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Grasp Diffusion Network: Learning Grasp Generators from Partial Point Clouds with Diffusion Models in SO(3)xR3
  2. 2024DexGrasp-Diffusion: Diffusion-based Unified Functional Grasp Synthesis Method for Multi-Dexterous Robotic Hands
  3. 2025DiffusionRL: Efficient Training of Diffusion Policies for Robotic Grasping Using RL-Adapted Large-Scale Datasets
  4. 2026Grounding Generative Policies in Physics: Optimization-Guided Diffusion for Robot Control
  5. 2024Diffusion-Informed Probabilistic Contact Search for Multi-Finger Manipulation
  6. 2026GraspGen-X: Cross-Embodiment 6-DOF Diffusion-based Grasping
  7. 2025Variational Shape Inference for Grasp Diffusion on SE(3)
  8. 2024Joint Diffusion for Universal Hand-Object Grasp Generation