# SceneWeaver: All-in-One 3D Scene Synthesis with an Extensible and Self-Reflective Agent

arXiv: https://arxiv.org/abs/2509.20414 · HTML: https://mlsanity.com/paper/2509.20414/

## Key facts

- Cited by 56 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use.
- Builds on 33 of its 37 references that are in the map.
- First posted on arXiv 24 Sep 2025.

## Placed in

- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 30 of 33)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation](https://mlsanity.com/paper/2308.08155/) (2023)
- [Objaverse: A Universe of Annotated 3D Objects](https://mlsanity.com/paper/2212.08051/) (2022)
- [3D-FRONT: 3D Furnished Rooms with layOuts and semaNTics](https://mlsanity.com/paper/2011.09127/) (2020)
- [ProcTHOR: Large-Scale Embodied AI Using Procedural Generation](https://mlsanity.com/paper/2206.06994/) (2022)
- [LayoutGPT: Compositional Visual Planning and Generation with Large Language Models](https://mlsanity.com/paper/2305.15393/) (2023)
- [3D-FUTURE: 3D Furniture shape with TextURE](https://mlsanity.com/paper/2009.09633/) (2020)
- [Holodeck: Language Guided Generation of 3D Embodied AI Environments](https://mlsanity.com/paper/2312.09067/) (2023)
- [Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Models](https://mlsanity.com/paper/2406.09403/) (2024)
- [AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environments](https://mlsanity.com/paper/2405.07960/) (2024)
- [ATISS: Autoregressive Transformers for Indoor Scene Synthesis](https://mlsanity.com/paper/2110.03675/) (2021)
- [OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understanding](https://mlsanity.com/paper/2305.10764/) (2023)
- [Habitat Synthetic Scenes Dataset (HSSD-200): An Analysis of 3D Scene Scale and Realism Tradeoffs for ObjectGoal Navigation](https://mlsanity.com/paper/2306.11290/) (2023)
- [DiffuScene: Denoising Diffusion Models for Generative Indoor Scene Synthesis](https://mlsanity.com/paper/2303.14207/) (2023)
- [GPT-4V(ision) is a Human-Aligned Evaluator for Text-to-3D Generation](https://mlsanity.com/paper/2401.04092/) (2024)
- [Infinigen Indoors: Photorealistic Indoor Scenes using Procedural Generation](https://mlsanity.com/paper/2406.11824/) (2024)
- [SceneCraft: An LLM Agent for Synthesizing 3D Scene as Blender Code](https://mlsanity.com/paper/2403.01248/) (2024)
- [PhyScene: Physically Interactable 3D Scene Synthesis for Embodied AI](https://mlsanity.com/paper/2404.09465/) (2024)
- [GALA3D: Towards Text-to-3D Complex Scene Generation via Layout-guided Generative Gaussian Splatting](https://mlsanity.com/paper/2402.07207/) (2024)
- [LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Models](https://mlsanity.com/paper/2412.02193/) (2024)
- [Automated Creation of Digital Cousins for Robust Policy Learning](https://mlsanity.com/paper/2410.07408/) (2024)
- [I-Design: Personalized LLM Interior Designer](https://mlsanity.com/paper/2404.02838/) (2024)
- [AnyHome: Open-Vocabulary Generation of Structured and Textured 3D Homes](https://mlsanity.com/paper/2312.06644/) (2023)
- [LEGO-Net: Learning Regular Rearrangements of Objects in Rooms](https://mlsanity.com/paper/2301.09629/) (2023)
- [Human-centric Indoor Scene Synthesis Using Stochastic Grammar](https://mlsanity.com/paper/1808.08473/) (2018)
- [OctoTools: An Agentic Framework with Extensible Tools for Complex Reasoning](https://mlsanity.com/paper/2502.11271/) (2025)
- [EchoScene: Indoor Scene Generation via Information Echo over Scene Graph Diffusion](https://mlsanity.com/paper/2405.00915/) (2024)
- [Scenethesis: A Language and Vision Agentic Framework for 3D Scene Generation](https://mlsanity.com/paper/2505.02836/) (2025)
- [SceneTex: High-Quality Texture Synthesis for Indoor Scenes via Diffusion Priors](https://mlsanity.com/paper/2311.17261/) (2023)
- [Architect: Generating Vivid and Interactive 3D Scenes with Hierarchical 2D Inpainting](https://mlsanity.com/paper/2411.09823/) (2024)

## Built on by (most cited first, 30 of 56)

- [SAGE: Scalable Agentic 3D Scene Generation for Embodied AI](https://mlsanity.com/paper/2602.10116/) (2026)
- [RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies](https://mlsanity.com/paper/2604.09860/) (2026)
- [Toward Personalized LLM-Powered Agents: Foundations, Evaluation, and Future Directions](https://mlsanity.com/paper/2602.22680/) (2026)
- [MANSION: Multi-floor lANguage-to-3D Scene generatIOn for loNg-horizon tasks](https://mlsanity.com/paper/2603.11554/) (2026)
- [Agentic Real2Sim: Physics-based World Modeling with Vision-Language Agents](https://mlsanity.com/paper/2607.19190/) (2026)
- [WorldClaw: Agentic 3D Open-World Generation at Scale](https://mlsanity.com/paper/2608.05248/) (2026)
- [I-Scene: 3D Instance Models are Implicit Generalizable Spatial Learners](https://mlsanity.com/paper/2512.13683/) (2025)
- [SceneReVis: A Self-Reflective Vision-Grounded Framework for 3D Indoor Scene Synthesis via Multi-Turn RL](https://mlsanity.com/paper/2602.09432/) (2026)
- [SceneAssistant: A Visual Feedback Agent for Open-Vocabulary 3D Scene Generation](https://mlsanity.com/paper/2603.12238/) (2026)
- [InfiniBench: Infinite Benchmarking for Visual Spatial Reasoning with Customizable Scene Complexity](https://mlsanity.com/paper/2511.18200/) (2025)
- [Code-as-Room: Generating 3D Rooms from Top-Down View Images via Agentic Code Synthesis](https://mlsanity.com/paper/2605.18451/) (2026)
- [ArtiWorld: LLM-Driven Articulation of 3D Objects in Scenes](https://mlsanity.com/paper/2511.12977/) (2025)
- [UrbanWorld2.0: A Multimodal Agentic Framework for Reality-Aligned 3D World Generation at City-Scale](https://mlsanity.com/paper/2511.18005/) (2025)
- [Edit-As-Act: Goal-Regressive Planning for Open-Vocabulary 3D Indoor Scene Editing](https://mlsanity.com/paper/2603.17583/) (2026)
- [StoryBlender: Inter-Shot Consistent and Editable 3D Storyboard with Spatial-temporal Dynamics](https://mlsanity.com/paper/2604.03315/) (2026)
- [STABLE: Simulation-Ready Tabletop Layout Generation via a Semantics-Physics Dual System](https://mlsanity.com/paper/2605.16137/) (2026)
- [MUSE: Agentic 3D Scene Authoring via Memory-Grounded Incremental Requirement Satisfaction](https://mlsanity.com/paper/2606.14168/) (2026)
- [SceneNAT: Masked Generative Modeling for Language-Guided Indoor Scene Synthesis](https://mlsanity.com/paper/2601.07218/) (2026)
- [SceneOrchestra: Efficient Agentic 3D Scene Synthesis via Full Tool-Call Trajectory Generation](https://mlsanity.com/paper/2604.19907/) (2026)
- [WorldAct: Activating Monolithic 3D Worlds into Interactive-Ready Object-Centric Scenes](https://mlsanity.com/paper/2605.15843/) (2026)
- [SceneConductor: 3D Scene Generation from a Single Image with Multi-Agent Orchestration](https://mlsanity.com/paper/2606.08402/) (2026)
- [ShellMaker: Language-Guided Exterior Completion under Structural Constraints](https://mlsanity.com/paper/2606.31680/) (2026)
- [VibeWorlding: Can Multimodal Agents Construct 3D Open Worlds End-to-End?](https://mlsanity.com/paper/2608.15265/) (2026)
- [GeoSceneGraph: Geometric Scene Graph Diffusion Model for Text-guided 3D Indoor Scene Synthesis](https://mlsanity.com/paper/2511.14884/) (2025)
- [Extended to Reality: Prompt Injection in 3D Environments](https://mlsanity.com/paper/2602.07104/) (2026)
- [Seen2Scene: Completing Realistic 3D Scenes with Visibility-Guided Flow](https://mlsanity.com/paper/2603.28548/) (2026)
- [Co-generation of Layout and Shape from Text via Autoregressive 3D Diffusion](https://mlsanity.com/paper/2604.16552/) (2026)
- [3D Generation for Embodied AI and Robotic Simulation: A Survey](https://mlsanity.com/paper/2604.26509/) (2026)
- [Function2Scene: 3D Indoor Scene Layout from Functional Specifications](https://mlsanity.com/paper/2605.30819/) (2026)
- [iARCS: Iterative Agentic RL for Controllable 3D Scene Generation](https://mlsanity.com/paper/2608.06161/) (2026)

## Abstract (arXiv)

Indoor scene synthesis has become increasingly important with the rise of Embodied AI, which requires 3D environments that are not only visually realistic but also physically plausible and functionally diverse. While recent approaches have advanced visual fidelity, they often remain constrained to fixed scene categories, lack sufficient object-level detail and physical consistency, and struggle to align with complex user instructions. In this work, we present SceneWeaver, a reflective agentic framework that unifies diverse scene synthesis paradigms through tool-based iterative refinement. At its core, SceneWeaver employs a language model-based planner to select from a suite of extensible scene generation tools, ranging from data-driven generative models to visual- and LLM-based methods, guided by self-evaluation of physical plausibility, visual realism, and semantic alignment with user input. This closed-loop reason-act-reflect design enables the agent to identify semantic inconsistencies, invoke targeted tools, and update the environment over successive iterations. Extensive experiments on both common and open-vocabulary room types demonstrate that SceneWeaver not only outperforms prior methods on physical, visual, and semantic metrics, but also generalizes effectively to complex scenes with diverse instructions, marking a step toward general-purpose 3D environment generation. Project website: https://scene-weaver.github.io/.