mlsanityData to 7 Oct 2026

Scenethesis: A Language and Vision Agentic Framework for 3D Scene Generation

arXiv 2505.02836 ↗First posted 5 May 2025cs.CVcited by 65

On the map: LLM agents and tool use → Multimodal and embodied agents

Abstract (arXiv)

Synthesizing interactive 3D scenes from text is essential for gaming, virtual reality, and embodied AI. However, existing methods face several challenges. Learning-based approaches depend on small-scale indoor datasets, limiting the scene diversity and layout complexity. While large language models (LLMs) can leverage diverse text-domain knowledge, they struggle with spatial realism, often producing unnatural object placements that fail to respect common sense. Our key insight is that vision perception can bridge this gap by providing realistic spatial guidance that LLMs lack. To this end, we introduce Scenethesis, a training-free agentic framework that integrates LLM-based scene planning with vision-guided layout refinement. Given a text prompt, Scenethesis first employs an LLM to draft a coarse layout. A vision module then refines it by generating an image guidance and extracting scene structure to capture inter-object relations. Next, an optimization module iteratively enforces accurate pose alignment and physical plausibility, preventing artifacts like object penetration and instability. Finally, a judge module verifies spatial coherence. Comprehensive experiments show that Scenethesis generates diverse, realistic, and physically plausible 3D interactive scenes, making it valuable for virtual content creation, simulation environments, and embodied AI research.

Key facts. Cited by 65 papers in the map (data through 7 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 47 of its 55 references that are in the map. First posted on arXiv 5 May 2025.

Builds on

47 of 55 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024SceneCraft: An LLM Agent for Synthesizing 3D Scene as Blender Code2 Mar

Other references · 46

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  4. 2024GPT-4o System Cardcited by 4,467
  5. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  6. 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 914
  7. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  8. 2020Accelerating 3D Deep Learning with PyTorch3Dcited by 781
  9. 2023DL3DV-10K: A Large-Scale Scene Dataset for Deep Learning-based 3D Visioncited by 514
  10. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 465
  11. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 429
  12. 2024Depth Pro: Sharp Monocular Metric Depth in Less Than a Secondcited by 395
34 more
  1. 2024RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robotscited by 391
  2. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 369
  3. 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 337
  4. 2023ReconFusion: 3D Reconstruction with Diffusion Priorscited by 317
  5. 20203D-FUTURE: 3D Furniture shape with TextUREcited by 306
  6. 2023RoMa: Robust Dense Feature Matchingcited by 296
  7. 2023LucidDreamer: Towards High-Fidelity Text-to-3D Generation via Interval Score Matchingcited by 290
  8. 2023Holodeck: Language Guided Generation of 3D Embodied AI Environmentscited by 257
  9. 2020ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulationcited by 249
  10. 2023Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Modelscited by 245
  11. 2023RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulationcited by 206
  12. 2024WonderWorld: Interactive 3D Scene Generation from a Single Imagecited by 197
  13. 2021ATISS: Autoregressive Transformers for Indoor Scene Synthesiscited by 195
  14. 2023Habitat Synthetic Scenes Dataset (HSSD-200): An Analysis of 3D Scene Scale and Realism Tradeoffs for ObjectGoal Navigationcited by 162
  15. 2023SceneScape: Text-Driven Consistent Scene Generationcited by 160
  16. 2023DiffuScene: Denoising Diffusion Models for Generative Indoor Scene Synthesiscited by 159
  17. 2023WonderJourney: Going from Anywhere to Everywherecited by 157
  18. 2020SceneFormer: Indoor Scene Generation with Transformerscited by 154
  19. 2024Infinigen Indoors: Photorealistic Indoor Scenes using Procedural Generationcited by 129
  20. 2023Text2NeRF: Text-Driven 3D Scene Generation with Neural Radiance Fieldscited by 122
  21. 2024PhyScene: Physically Interactable 3D Scene Synthesis for Embodied AIcited by 116
  22. 2024InstructScene: Instruction-Driven 3D Indoor Scene Synthesis with Semantic Graph Priorcited by 110
  23. 2024GALA3D: Towards Text-to-3D Complex Scene Generation via Layout-guided Generative Gaussian Splattingcited by 104
  24. 2024LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Modelscited by 94
  25. 2023Telling Left from Right: Identifying Geometry-Aware Semantic Correspondencecited by 85
  26. 2019SceneGraphNet: Neural Message Passing for 3D Indoor Scene Augmentationcited by 55
  27. 2020End-to-End Optimization of Scene Layoutcited by 52
  28. 2023SecondPose: SE(3)-Consistent Dual-Stream Feature Fusion for Category-Level Pose Estimationcited by 52
  29. 2024Open-Universe Indoor Scene Generation using LLM Program Synthesis and Uncurated Object Databasescited by 43
  30. 2024SceneTeller: Language-to-3D Scene Generationcited by 43
  31. 2024Architect: Generating Vivid and Interactive 3D Scenes with Hierarchical 2D Inpaintingcited by 37
  32. 2024Zero-Shot Scene Reconstruction from Single Images with Deep Prior Assemblycited by 34
  33. 2022Phone2Proc: Bringing Robust Robots Into Our Chaotic Worldcited by 24
  34. 2024Atlas3D: Physically Constrained Self-Supporting Text-to-3D for Simulation and Fabricationcited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 24; 2026: 41.

65 papers.

Sort:

By area:

  1. 2025SceneWeaver: All-in-One 3D Scene Synthesis with an Extensible and Self-Reflective Agentcited by 53
  2. 2025SceneGen: Single-Image 3D Scene Generation in One Feedforward Passcited by 49
  3. 2025World-in-World: World Models in a Closed-Loop Worldcited by 44
  4. 2025VLM4D: Towards Spatiotemporal Awareness in Vision Language Modelscited by 43
  5. 2026Vision-as-Inverse-Graphics Agent via Interleaved Multimodal Reasoningcited by 40
  6. 2026SAGE: Scalable Agentic 3D Scene Generation for Embodied AIcited by 40
  7. 2025Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarkscited by 31
  8. 2025WorldGen: From Text to Traversable and Interactive 3D Worldscited by 18
  9. 2025MesaTask: Towards Task-Driven Tabletop Scene Generation via 3D Spatial Reasoningcited by 15
  10. 2025HoloScene: Simulation-Ready Interactive 3D Worlds from a Single Videocited by 13
  11. 20263DCodeBench: Benchmarking Agentic Procedural 3D Modeling Via Codecited by 9
  12. 2025Yo'City: Personalized and Boundless 3D Realistic City Scene Generation via Self-Critic Expansioncited by 8
38 more
  1. 2025Video Perception Models for 3D Scene Synthesiscited by 7
  2. 2026CausalSpatial: A Benchmark for Object-Centric Causal Spatial Reasoningcited by 7
  3. 2026PhyScensis: Physics-Augmented LLM Agents for Complex Physical Scene Arrangementcited by 7
  4. 2025Vision-Language Models as Differentiable Semantic and Spatial Rewards for Text-to-3D Generationcited by 6
  5. 2025I-Scene: 3D Instance Models are Implicit Generalizable Spatial Learnerscited by 6
  6. 2026SceneReVis: A Self-Reflective Vision-Grounded Framework for 3D Indoor Scene Synthesis via Multi-turn RLcited by 6
  7. 2026WorldCoder-Bench: Benchmarking Physically Grounded 3D World Synthesiscited by 6
  8. 2026WorldClaw: Agentic 3D Open-World Generation at Scalecited by 6
  9. 2025ImmerseGen: Agent-Guided Immersive World Generation with Alpha-Textured Proxiescited by 5
  10. 2025DisCo-Layout: Disentangling and Coordinating Semantic and Physical Refinement in a Multi-Agent Framework for 3D Indoor Layout Synthesiscited by 5
  11. 2025InfiniBench: Infinite Benchmarking for Visual Spatial Reasoning with Customizable Scene Complexitycited by 5
  12. 2026SceneAssistant: A Visual Feedback Agent for Open-Vocabulary 3D Scene Generationcited by 5
  13. 2025RobotSmith: Generative Robotic Tool Design for Acquisition of Complex Manipulation Skillscited by 4
  14. 2026FlowScene: Style-Consistent Indoor Scene Generation with Multimodal Graph Rectified Flowcited by 4
  15. 2026SimuScene: Simulation-Ready Compositional 3D Scene Reconstruction from a Single Imagecited by 4
  16. 2026Rein3D: Reinforced 3D Indoor Scene Generation with Panoramic Video Diffusion Modelscited by 3
  17. 2025GACO-CAD: Geometry-Augmented and Conciseness-Optimized CAD Model Generation from Single Imagecited by 2
  18. 2025TabletopGen: Tabletop Scene Generation and Interactive Simulation for Robotic Manipulationcited by 2
  19. 2026Mind-of-Director: Multi-modal Agent-Driven Film Previsualization via Collaborative Decision-Makingcited by 2
  20. 2026SceneCritic: A Symbolic Evaluator for 3D Indoor Scene Synthesiscited by 2
  21. 2026SceneOrchestra: Efficient Agentic 3D Scene Synthesis via Full Tool-Call Trajectory Generationcited by 2
  22. 2026ArchSIBench: Benchmarking the Architectural Spatial Intelligence of Vision-Language Modelscited by 2
  23. 2026SceneConductor: 3D Scene Generation from a Single Image with Multi-Agent Orchestrationcited by 2
  24. 2026NaLA: A 3D Native LLM Layout Agent for High-quality 3D Scene Generationcited by 2
  25. 2026One Video, One World: Turning Monocular Video into Physical 4D Scenescited by 2
  26. 2025HiGS: Hierarchical Generative Scene Framework for Multi-Step Associative Semantic Spatial Compositioncited by 1
  27. 2025Apples on the Table? Evaluating Text-Guided 3D Scene Synthesis via Fine-Grained Constraint Verificationcited by 1
  28. 2025GeoSceneGraph: Geometric Scene Graph Diffusion Model for Text-guided 3D Indoor Scene Synthesiscited by 1
  29. 20263D Space as a Scratchpad for Editable Text-to-Image Generationcited by 1
  30. 2026Co-generation of Layout and Shape from Text via Autoregressive 3D Diffusioncited by 1
  31. 2026Perceive-then-Plan: Layout-as-Policy for Monocular 3D Scene Layout Estimationcited by 1
  32. 2026PhyScene3D: Physically Consistent Interactive 3D Tabletop Scene Generationcited by 1
  33. 2026ViSculpt: Visual-Centric Agentic Geometry Editingcited by 1
  34. 2026GIF: Agentic Generation of Interactive and Functional Object Compositions for Robot Learningcited by 1
  35. 2025Spatial Knowledge Graph-Guided Multimodal Synthesiscited by 0
  36. 2025Handle-based Mesh Deformation Guided By Vision Language Modelcited by 0
  37. 2025LLM-BT-Terms: Back-Translation as a Framework for Terminology Standardization and Dynamic Semantic Embeddingcited by 0
  38. 2026Seen2Scene: Completing Realistic 3D Scenes with Visibility-Guided Flowcited by 0

50 most cited of 65 All 65 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026SceneMosaic: Efficient and Diverse Simulation-Ready Scene Generation via Hybrid Agentic Layout Evolution
  2. 2026SceneSmith: Agentic Generation of Simulation-Ready Indoor Scenes
  3. 2026SceneFoundry: Generating Interactive Infinite 3D Worlds
  4. 2025LayoutAgent: A Vision-Language Agent Guided Compositional Diffusion for Spatial Layout Planning
  5. 2026DesignAgent3D: Interactive 3D Scene Editing via Designer-like Multimodal Reasoning
  6. 2026SceneCode: Executable World Programs for Editable Indoor Scenes with Articulated Objects
  7. 2026SimWorlds: A Multi-Agent System for Dynamic 3D Scene Creation
  8. 2026HOG-Layout: Hierarchical 3D Scene Generation, Optimization and Editing via Vision-Language Models