# SceneSmith: Agentic Generation of Simulation-Ready Indoor Scenes

arXiv: https://arxiv.org/abs/2602.09153 · HTML: https://mlsanity.com/paper/2602.09153/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use.
- Builds on 19 of its 46 references that are in the map.
- First posted on arXiv 9 Feb 2026.

## Placed in

- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 19 of 19)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [π0.5: a Vision-Language-Action Model with Open-World Generalization](https://mlsanity.com/paper/2504.16054/) (2025)
- [Objaverse: A Universe of Annotated 3D Objects](https://mlsanity.com/paper/2212.08051/) (2022)
- [SAM 3: Segment Anything with Concepts](https://mlsanity.com/paper/2511.16719/) (2025)
- [Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V](https://mlsanity.com/paper/2310.11441/) (2023)
- [SAM 3D: 3Dfy Anything in Images](https://mlsanity.com/paper/2511.16624/) (2025)
- [Native and Compact Structured Latents for 3D Generation](https://mlsanity.com/paper/2512.14692/) (2025)
- [Habitat Synthetic Scenes Dataset (HSSD-200): An Analysis of 3D Scene Scale and Realism Tradeoffs for ObjectGoal Navigation](https://mlsanity.com/paper/2306.11290/) (2023)
- [A Careful Examination of Large Behavior Models for Multitask Dexterous Manipulation](https://mlsanity.com/paper/2507.05331/) (2025)
- [URDFormer: A Pipeline for Constructing Articulated Simulation Environments from Real-World Images](https://mlsanity.com/paper/2405.11656/) (2024)
- [LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Models](https://mlsanity.com/paper/2412.02193/) (2024)
- [SINGAPO: Single Image Controlled Generation of Articulated Parts in Objects](https://mlsanity.com/paper/2410.16499/) (2024)
- [Mixed Diffusion for 3D Indoor Scene Synthesis](https://mlsanity.com/paper/2405.21066/) (2024)
- [A Systematic Study of Data Modalities and Strategies for Co-training Large Behavior Models for Robot Manipulation](https://mlsanity.com/paper/2602.01067/) (2026)
- [LL3M: Large Language 3D Modelers](https://mlsanity.com/paper/2508.08228/) (2025)
- [ArtVIP: Articulated Digital Assets of Visual Realism, Modular Interaction, and Physical Fidelity for Robot Learning](https://mlsanity.com/paper/2506.04941/) (2025)
- [RoomCraft: Controllable and Complete 3D Indoor Scene Generation](https://mlsanity.com/paper/2506.22291/) (2025)
- [3D-Generalist: Self-Improving Vision-Language-Action Models for Crafting 3D Worlds](https://mlsanity.com/paper/2507.06484/) (2025)

## Built on by (most cited first, 30 of 46)

- [Cosmos 3: Omnimodal World Models for Physical AI](https://mlsanity.com/paper/2606.02800/) (2026)
- [SimWorld Studio: Automatic Environment Generation with Evolving Coding Agent for Embodied Agent Learning](https://mlsanity.com/paper/2605.09423/) (2026)
- [Not All Features Are Created Equal: A Mechanistic Study of Vision-Language-Action Models](https://mlsanity.com/paper/2603.19233/) (2026)
- [SceneCode: Executable World Programs for Editable Indoor Scenes with Articulated Objects](https://mlsanity.com/paper/2605.19587/) (2026)
- [Agentic Real2Sim: Physics-based World Modeling with Vision-Language Agents](https://mlsanity.com/paper/2607.19190/) (2026)
- [WorldClaw: Agentic 3D Open-World Generation at Scale](https://mlsanity.com/paper/2608.05248/) (2026)
- [REST3D: Reconstructing Physically Stable 3D Scenes from a Single Image](https://mlsanity.com/paper/2605.30338/) (2026)
- [Engine-Native Editable 3D World Reconstruction with Objects and Lighting](https://mlsanity.com/paper/2607.20889/) (2026)
- [Code-as-Room: Generating 3D Rooms from Top-Down View Images via Agentic Code Synthesis](https://mlsanity.com/paper/2605.18451/) (2026)
- [PairCoder++: Pair Programming as a Universal Paradigm for Verified Code-Driven Multimodal and Structured-Artifact Generation](https://mlsanity.com/paper/2607.01883/) (2026)
- [NymeriaPlus: Enriching Nymeria Dataset with Additional Annotations and Data](https://mlsanity.com/paper/2603.18496/) (2026)
- [MUSE: Agentic 3D Scene Authoring via Memory-Grounded Incremental Requirement Satisfaction](https://mlsanity.com/paper/2606.14168/) (2026)
- [FIRE3D: Feed-forward Interactive 3D Scene Reconstruction Within A Minute](https://mlsanity.com/paper/2609.08848/) (2026)
- [SceneOrchestra: Efficient Agentic 3D Scene Synthesis via Full Tool-Call Trajectory Generation](https://mlsanity.com/paper/2604.19907/) (2026)
- [Beyond Placement and Articulation: Usage-Driven Code Scenes for Embodied Interaction](https://mlsanity.com/paper/2608.18840/) (2026)
- [Seen2Scene: Completing Realistic 3D Scenes with Visibility-Guided Flow](https://mlsanity.com/paper/2603.28548/) (2026)
- [3D Generation for Embodied AI and Robotic Simulation: A Survey](https://mlsanity.com/paper/2604.26509/) (2026)
- [Aggregating LLM-Based Weak Verifiers for Spatial Layout Generation](https://mlsanity.com/paper/2606.05268/) (2026)
- [FuncRoom-Agent: Sequential Feed-Forward 3D Functional Indoor Scene Generation](https://mlsanity.com/paper/2608.29519/) (2026)
- [GIF: Agentic Generation of Interactive and Functional Object Compositions for Robot Learning](https://mlsanity.com/paper/2609.05927/) (2026)
- [Agentic Visual Generation: From Generative Models to Agentic Control](https://mlsanity.com/paper/2609.06758/) (2026)
- [DiagGen: Agentic Generation of Deformable Assets with Sim-based Diagnostics for Robotic Simulation](https://mlsanity.com/paper/2609.23103/) (2026)
- [ARSTAG: An Agentic Real2Sim2Real System for Task-Specific Robot Data Generation](https://mlsanity.com/paper/2609.24563/) (2026)
- [HARMONY: Hierarchical Agentic Reasoning for MONocular Image-to-Scene Synthesis](https://mlsanity.com/paper/2609.26793/) (2026)
- [Self-Evolving Coding Agents: From Digital Programs to Physical-World Intelligence](https://mlsanity.com/paper/2609.35432/) (2026)
- [Code4Scene: Benchmarking Coding Agents for Constructing and Editing 3D Scenes](https://mlsanity.com/paper/2609.36777/) (2026)
- [GenPHRI: Agentic Generative Simulation for Physical Human-Robot Interaction](https://mlsanity.com/paper/2604.08664/) (2026)
- [VoxScene: Anchor-Conditioned Voxel Diffusion for Indoor Scene Arrangement](https://mlsanity.com/paper/2605.17102/) (2026)
- [OSMa-Bench++: Toward Open-Ended Benchmarking of Semantic Mapping for Manipulation with Prompt-Generated Synthetic Scenes](https://mlsanity.com/paper/2605.26831/) (2026)
- [Architect-Ant: Editable Automatic Furnishing of Architectural Floor Plans](https://mlsanity.com/paper/2606.10953/) (2026)

## Abstract (arXiv)

Simulation has become a key tool for training and evaluating home robots at scale, yet existing environments fail to capture the diversity and physical complexity of real indoor spaces. Current scene synthesis methods produce sparsely furnished rooms that lack the dense clutter, articulated furniture, and physical properties essential for robotic manipulation. We introduce SceneSmith, a hierarchical agentic framework that generates simulation-ready indoor environments from natural language prompts. SceneSmith constructs scenes through successive stagesunicodex2013from architectural layout to furniture placement to small object populationunicodex2013each implemented as an interaction among VLM agents: designer, critic, and orchestrator. The framework tightly integrates asset generation through text-to-3D synthesis for static objects, dataset retrieval for articulated objects, and physical property estimation. SceneSmith generates 3-6x more objects than prior methods, with <2% inter-object collisions and 96% of objects remaining stable under physics simulation. In a user study with 205 participants, it achieves 92% average realism and 91% average prompt faithfulness win rates against baselines. We further demonstrate that these environments can be used in an end-to-end pipeline for automatic robot policy evaluation.