mlsanityData to 7 Oct 2026

Diffusion-based Generation, Optimization, and Planning in 3D Scenes

arXiv 2301.06015 ↗First posted 15 Jan 2023cs.CVcited by 237

On the map: Diffusion models → Diffusion and flow policies for robotics

Abstract (arXiv)

We introduce SceneDiffuser, a conditional generative model for 3D scene understanding. SceneDiffuser provides a unified model for solving scene-conditioned generation, optimization, and planning. In contrast to prior works, SceneDiffuser is intrinsically scene-aware, physics-based, and goal-oriented. With an iterative sampling strategy, SceneDiffuser jointly formulates the scene-aware generation, physics-based optimization, and goal-oriented planning via a diffusion-based denoising process in a fully differentiable fashion. Such a design alleviates the discrepancies among different modules and the posterior collapse of previous scene-conditioned generative models. We evaluate SceneDiffuser with various 3D scene understanding tasks, including human pose and motion generation, dexterous grasp generation, path planning for 3D navigation, and motion planning for robot arms. The results show significant improvements compared with previous models, demonstrating the tremendous potential of SceneDiffuser for the broad community of 3D scene understanding.

Key facts. Cited by 237 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 81 of its 99 references that are in the map. First posted on arXiv 15 Jan 2023.

Builds on

81 of 99 references.

References · 81

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2020Denoising Diffusion Implicit Modelscited by 9,655
  7. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  10. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  11. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  12. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
38 more
  1. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  2. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  3. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  4. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  5. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  6. 2020Point Transformercited by 1,462
  7. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  8. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,155
  9. 2022Human Motion Diffusion Modelcited by 1,062
  10. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  11. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  12. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 882
  13. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  14. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  15. 2022MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Modelcited by 768
  16. 2021Diffusion Probabilistic Models for 3D Point Cloud Generationcited by 734
  17. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 501
  18. 2020Object Goal Navigation using Goal-Oriented Semantic Explorationcited by 421
  19. 2020GRAB: A Dataset of Whole-Body Human Grasping of Objectscited by 397
  20. 2017Wasserstein Auto-Encoderscited by 371
  21. 2019DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Framescited by 352
  22. 2016Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learningcited by 351
  23. 2021HuMoR: 3D Human Motion Model for Robust Pose Estimationcited by 319
  24. 2020Learning to Explore using Active Neural SLAMcited by 308
  25. 2022SQA3D: Situated Question Answering in 3D Scenescited by 303
  26. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 300
  27. 2019Resolving 3D Human Pose Ambiguities with 3D Scene Constraintscited by 294
  28. 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 228
  29. 2021MP3: A Unified Model to Map, Perceive, Predict and Plancited by 211
  30. 2021Stochastic Scene-Aware Motion Predictioncited by 203
  31. 2021ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrationscited by 193
  32. 2019Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishingcited by 191
  33. 2021iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Taskscited by 191
  34. 2020DLow: Diversifying Latent Flows for Diverse Human Motion Predictioncited by 184
  35. 2019Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Taskscited by 177
  36. 2020Populating 3D Scenes by Learning Human-Scene Interactioncited by 164
  37. 2021Hand-Object Contact Consistency Reasoning for Human Grasps Generationcited by 160
  38. 2022HUMANISE: Language-conditioned Human Motion Generation in 3D Scenescited by 156

50 most cited of 81 All 81 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 44; 2024: 87; 2025: 71; 2026: 35.

237 papers.

Sort:

By area:

  1. 2024Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusioncited by 634
  2. 2023OmniControl: Control Any Joint at Any Time for Human Motion Generationcited by 233
  3. 2023Object Motion Guided Human Motion Synthesiscited by 206
  4. 2023InterDiff: Generating 3D Human-Object Interactions with Physics-Informed Diffusioncited by 181
  5. 2025History-Guided Video Diffusioncited by 165
  6. 2023CG-HOI: Contact-Guided 3D Human-Object Interaction Generationcited by 137
  7. 2024Scaling Up Dynamic Human-Scene Interaction Modelingcited by 137
  8. 2024SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understandingcited by 130
  9. 2024PhyScene: Physically Interactable 3D Scene Synthesis for Embodied AIcited by 116
  10. 2023Controllable Human-Object Interaction Synthesiscited by 107
  11. 2024CLoSD: Closing the Loop between Simulation and Diffusion for multi-task character controlcited by 103
  12. 2023Optimizing Diffusion Noise Can Serve As Universal Motion Priorscited by 97
38 more
  1. 2023Human Motion Generation: A Surveycited by 95
  2. 2023Unified Human-Scene Interaction via Prompted Chain-of-Contactscited by 91
  3. 2023HOI-Diff: Text-Driven Synthesis of 3D Human-Object Interactions using Diffusion Modelscited by 89
  4. 2024Multi-Track Timeline Control for Text-Driven 3D Human Motion Generationcited by 88
  5. 2024Generating Human Interaction Motions in Scenes with Text Controlcited by 80
  6. 2024Move as You Say, Interact as You Can: Language-guided Human Motion Generation with Scene Affordancecited by 79
  7. 2023ADAPT: Action-aware Driving Caption Transformercited by 78
  8. 2023NIFTY: Neural Object Interaction Fields for Guided Human Motion Synthesiscited by 78
  9. 2024Rethinking Diffusion for Text-Driven Human Motion Generation: Redundant Representations, Evaluation, and Masked Autoregressioncited by 78
  10. 2023ARNOLD: A Benchmark for Language-Grounded Task Learning With Continuous States in Realistic 3D Scenescited by 68
  11. 2024CoMo: Controllable Motion Generation through Language Guided Pose Code Editingcited by 66
  12. 2024Large Motion Model for Unified Multi-Modal Motion Generationcited by 65
  13. 20253D and 4D World Modeling: A Surveycited by 64
  14. 2024InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interactioncited by 63
  15. 2023ReMoS: 3D Motion-Conditioned Reaction Synthesis for Two-Person Interactionscited by 62
  16. 2024Human-Object Interaction from Human-Level Instructionscited by 62
  17. 2024DiffH2O: Diffusion-Based Synthesis of Hand-Object Interactions from Textual Descriptionscited by 60
  18. 2024Lodge: A Coarse to Fine Diffusion Network for Long Dance Generation Guided by the Characteristic Dance Primitivescited by 59
  19. 2023Enhancing Deep Reinforcement Learning: A Tutorial on Generative Diffusion Models in Network Optimizationcited by 56
  20. 2024Autonomous Character-Scene Interaction Synthesis from Text Instructioncited by 56
  21. 2023CAMS: CAnonicalized Manipulation Spaces for Category-Level Functional Hand-Object Manipulation Synthesiscited by 54
  22. 2023SceneWiz3D: Towards Text-guided 3D Scene Compositioncited by 54
  23. 2023ArtiGrasp: Physically Plausible Synthesis of Bi-Manual Dexterous Grasping and Articulationcited by 53
  24. 2023Interactive Character Control with Auto-Regressive Motion Diffusion Modelscited by 52
  25. 2023Physically Plausible Full-Body Hand-Object Interaction Synthesiscited by 52
  26. 2024Streaming Diffusion Policy: Fast Policy Synthesis with Variable Noise Diffusion Modelscited by 51
  27. 2023LEMON: Learning 3D Human-Object Interaction Relation from 2D Imagescited by 49
  28. 2024Generating Human Motion in 3D Scenes from Text Descriptionscited by 49
  29. 2023Synthesizing Physically Plausible Human Motions in 3D Scenescited by 47
  30. 2024RoHM: Robust Human Motion Reconstruction via Diffusioncited by 46
  31. 2024DexDiffuser: Generating Dexterous Grasps with Diffusion Modelscited by 46
  32. 2024THOR: Text to Human-Object Interaction Diffusion via Relation Interventioncited by 46
  33. 2025TokenHSI: Unified Synthesis of Physical Human-Scene Interactions through Task Tokenizationcited by 46
  34. 2024PhyRecon: Physically Plausible Neural Scene Reconstructioncited by 44
  35. 2023Generative Proxemics: A Prior for 3D Social Interaction from Imagescited by 43
  36. 2023Hierarchical Generation of Human-Object Interactions with Diffusion Probabilistic Modelscited by 43
  37. 2024PoCo: Policy Composition from and for Heterogeneous Robot Learningcited by 42
  38. 2023Detecting Human-Object Contact in Imagescited by 41

50 most cited of 237 All 237 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Steerable Scene Generation with Post Training and Inference-Time Search
  2. 2026SceneFoundry: Generating Interactive Infinite 3D Worlds
  3. 2026GeomVLA: Unifying Scene, Motion, and Action in 3D
  4. 2023DiffuScene: Denoising Diffusion Models for Generative Indoor Scene Synthesis
  5. 2026KnowDiffuser: A Knowledge-Guided Diffusion Planner with LLM Reasoning
  6. 2022RenderDiffusion: Image Diffusion for 3D Reconstruction, Inpainting and Generation
  7. 20243D Diffuser Actor: Policy Diffusion with 3D Scene Representations
  8. 2024SCENIC: Scene-aware Semantic Navigation with Instruction-guided Control