mlsanityData to 7 Oct 2026

PhyScene: Physically Interactable 3D Scene Synthesis for Embodied AI

arXiv 2404.09465 ↗First posted 15 Apr 2024cs.CV cs.AI cs.LGcited by 116

Not on the map. Why?

Abstract (arXiv)

With recent developments in Embodied Artificial Intelligence (EAI) research, there has been a growing demand for high-quality, large-scale interactive scene generation. While prior methods in scene synthesis have prioritized the naturalness and realism of the generated scenes, the physical plausibility and interactivity of scenes have been largely left unexplored. To address this disparity, we introduce PhyScene, a novel method dedicated to generating interactive 3D scenes characterized by realistic layouts, articulated objects, and rich physical interactivity tailored for embodied agents. Based on a conditional diffusion model for capturing scene layouts, we devise novel physics- and interactivity-based guidance mechanisms that integrate constraints from object collision, room layout, and object reachability. Through extensive experiments, we demonstrate that PhyScene effectively leverages these guidance functions for physically interactable scene synthesis, outperforming existing state-of-the-art scene synthesis methods by a large margin. Our findings suggest that the scenes generated by PhyScene hold considerable potential for facilitating diverse skill acquisition among agents within interactive environments, thereby catalyzing further advancements in embodied AI research. Project website: http://physcene.github.io.

Key facts. Cited by 116 papers in the map (data through 7 Oct 2026). Builds on 57 of its 69 references that are in the map. First posted on arXiv 15 Apr 2024.

Builds on

57 of 69 references.

References · 57

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2020Denoising Diffusion Implicit Modelscited by 9,655
  3. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  4. 2022Classifier-Free Diffusion Guidancecited by 5,717
  5. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  6. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  7. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  8. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  9. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  10. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  11. 2018Demystifying MMD GANscited by 1,268
  12. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
38 more
  1. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  2. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 958
  3. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Modelscited by 924
  4. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  5. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  6. 2020SAPIEN: A SimulAted Part-based Interactive ENvironmentcited by 584
  7. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 501
  8. 2021ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Datacited by 442
  9. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 429
  10. 2023Universal Guidance for Diffusion Modelscited by 426
  11. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  12. 20203D-FRONT: 3D Furnished Rooms with layOuts and semaNTicscited by 375
  13. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 369
  14. 2022PhysDiff: Physics-Guided Human Motion Diffusion Modelcited by 365
  15. 2023An Embodied Generalist Agent in 3D Worldcited by 355
  16. 2023ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skillscited by 313
  17. 20203D-FUTURE: 3D Furniture shape with TextUREcited by 306
  18. 2019Resolving 3D Human Pose Ambiguities with 3D Scene Constraintscited by 294
  19. 2023Orbit: A Unified Simulation Framework for Interactive Robot Learning Environmentscited by 239
  20. 2023Diffusion-based Generation, Optimization, and Planning in 3D Scenescited by 237
  21. 2022MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generationcited by 225
  22. 2021ATISS: Autoregressive Transformers for Indoor Scene Synthesiscited by 195
  23. 2021ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrationscited by 193
  24. 2021iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Taskscited by 191
  25. 2020RoboTHOR: An Open Simulation-to-Real Embodied AI Platformcited by 190
  26. 2023Habitat Synthetic Scenes Dataset (HSSD-200): An Analysis of 3D Scene Scale and Realism Tradeoffs for ObjectGoal Navigationcited by 162
  27. 2023DiffuScene: Denoising Diffusion Models for Generative Indoor Scene Synthesiscited by 159
  28. 2020SceneFormer: Indoor Scene Generation with Transformerscited by 154
  29. 2023LayoutDM: Discrete Diffusion Model for Controllable Layout Generationcited by 141
  30. 2020Synthesizing Long-Term 3D Human Motion and Interaction in 3D Scenescited by 135
  31. 2022GAPartNet: Cross-Category Domain-Generalizable Object Perception and Manipulation via Generalizable and Actionable Partscited by 135
  32. 2024SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understandingcited by 130
  33. 2019IoU Loss for 2D/3D Object Detectioncited by 128
  34. 2021Graph-to-3D: End-to-End Generation and Manipulation of 3D Scenes Using Scene Graphscited by 83
  35. 2024Move as You Say, Interact as You Can: Language-guided Human Motion Generation with Scene Affordancecited by 79
  36. 2018Human-centric Indoor Scene Synthesis Using Stochastic Grammarcited by 73
  37. 2022VLMbench: A Compositional Benchmark for Vision-and-Language Manipulationcited by 68
  38. 2023ARNOLD: A Benchmark for Language-Grounded Task Learning With Continuous States in Realistic 3D Scenescited by 68

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 16; 2025: 47; 2026: 53.

116 papers.

Sort:

By area:

  1. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 175
  2. 2025CAST: Component-Aligned 3D Scene Reconstruction from an RGB Imagecited by 101
  3. 2025Sparc3D: Sparse Representation and Construction for High-Resolution 3D Shapes Modelingcited by 84
  4. 2025ArtGS: Building Interactable Replicas of Complex Articulated Objects via Gaussian Splattingcited by 67
  5. 2024SINGAPO: Single Image Controlled Generation of Articulated Parts in Objectscited by 66
  6. 2025Scenethesis: A Language and Vision Agentic Framework for 3D Scene Generationcited by 65
  7. 2025SceneWeaver: All-in-One 3D Scene Synthesis with an Extensible and Self-Reflective Agentcited by 53
  8. 2024Multi-modal Situated Reasoning in 3D Scenescited by 50
  9. 2024PhyRecon: Physically Plausible Neural Scene Reconstructioncited by 44
  10. 2025Articulate AnyMesh: Open-Vocabulary 3D Articulated Objects Modelingcited by 41
  11. 2026SAGE: Scalable Agentic 3D Scene Generation for Embodied AIcited by 40
  12. 2024Architect: Generating Vivid and Interactive 3D Scenes with Hierarchical 2D Inpaintingcited by 37
38 more
  1. 2024LLplace: The 3D Indoor Scene Layout Generation and Editing via Large Language Modelcited by 34
  2. 2025MetaScenes: Towards Automated Replica Creation for Real-world 3D Scanscited by 31
  3. 2025Decompositional Neural Scene Reconstruction with Generative Diffusion Priorcited by 30
  4. 2025Global-Local Tree Search in VLMs for 3D Indoor Scene Generationcited by 29
  5. 2024M2Diffuser: Diffusion-based Trajectory Optimization for Mobile Manipulation in 3D Scenescited by 26
  6. 2025SceneEval: Evaluating Semantic Coherence in Text-Conditioned 3D Indoor Scene Synthesiscited by 26
  7. 2025DreamArt: Generating Interactable Articulated Objects from a Single Imagecited by 25
  8. 2024PhysPart: Physically Plausible Part Completion for Interactable Objectscited by 24
  9. 2025Generative Physical AI in Vision: A Surveycited by 24
  10. 2025DSO: Aligning 3D Generators with Simulation Feedback for Physical Soundnesscited by 24
  11. 20253D Scene Generation: A Surveycited by 24
  12. 2025Generating Physically Stable and Buildable Brick Structures from Textcited by 23
  13. 2025HSM: Hierarchical Scene Motifs for Multi-Scale Indoor Scene Generationcited by 22
  14. 2024DeBaRA: Denoising-Based 3D Room Arrangement Generationcited by 20
  15. 2024Atlas3D: Physically Constrained Self-Supporting Text-to-3D for Simulation and Fabricationcited by 19
  16. 2025MMGDreamer: Mixed-Modality Graph for Geometry-Controllable 3D Indoor Scene Generationcited by 18
  17. 2025InternScenes: A Large-scale Simulatable Indoor Scene Dataset with Realistic Layoutscited by 18
  18. 2025CasaGPT: Cuboid Arrangement and Scene Assembly for Interior Designcited by 14
  19. 2025OptiScene: LLM-driven Indoor Scene Layout Generation via Scaled Human-aligned Data Synthesis and Multi-Stage Preference Optimizationcited by 14
  20. 2025Virtual Community: An Open World for Humans, Robots, and Societycited by 13
  21. 2024MOVIS: Enhancing Multi-Object Novel View Synthesis for Indoor Scenescited by 9
  22. 2025CHOrD: Generation of Collision-Free, House-Scale, and Organized Digital Twins for 3D Indoor Scenes with Controllable Floor Plans and Optimal Layoutscited by 9
  23. 2026SimRecon: SimReady Compositional Scene Reconstruction from Real Videoscited by 9
  24. 2025Steerable Scene Generation with Post Training and Inference-Time Searchcited by 8
  25. 2025FreeScene: Mixed Graph Diffusion for 3D Scene Synthesis from Free Promptscited by 8
  26. 2025Yo'City: Personalized and Boundless 3D Realistic City Scene Generation via Self-Critic Expansioncited by 8
  27. 2025Video Perception Models for 3D Scene Synthesiscited by 7
  28. 2025Robot Learning from Any Imagescited by 7
  29. 2025Seeing the Wind from a Falling Leafcited by 7
  30. 2026PhyScensis: Physics-Augmented LLM Agents for Complex Physical Scene Arrangementcited by 7
  31. 2024ROOT: VLM based System for Indoor Scene Understanding and Beyondcited by 6
  32. 2025SemLayoutDiff: Semantic Layout Generation with Diffusion Model for Indoor Scene Synthesiscited by 6
  33. 2025I-Scene: 3D Instance Models are Implicit Generalizable Spatial Learnerscited by 6
  34. 2026SceneReVis: A Self-Reflective Vision-Grounded Framework for 3D Indoor Scene Synthesis via Multi-turn RLcited by 6
  35. 2026WorldCoder-Bench: Benchmarking Physically Grounded 3D World Synthesiscited by 6
  36. 2024Human-Aware 3D Scene Generation with Spatially-constrained Diffusion Modelscited by 5
  37. 2024M3Bench: Benchmarking Whole-body Motion Generation for Mobile Manipulation in 3D Scenescited by 5
  38. 2025ReSpace: Text-Driven Autoregressive 3D Indoor Scene Synthesis and Editingcited by 5

50 most cited of 116 All 116 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026PhyScene3D: Physically Consistent Interactive 3D Tabletop Scene Generation
  2. 2026PhysForge: Generating Physics-Grounded 3D Assets for Interactive Virtual World
  3. 2024PhysDreamer: Physics-Based Interaction with 3D Objects via Video Generation
  4. 2026PhysMAS: Physics-Grounded Multi-Agent Synthesis of Compositional 4D Gaussians
  5. 2026SceneCode: Executable World Programs for Editable Indoor Scenes with Articulated Objects
  6. 2026NeoWorld-Pro: Programming Interactive Scenes from Monocular Images for Embodied Simulation
  7. 2026SimWorlds: A Multi-Agent System for Dynamic 3D Scene Creation
  8. 2026PhyGenHOI: Physically-Aware 4D Generation of Dynamic Human-Object Interactions