mlsanityData to 14 Oct 2026

Global-Local Tree Search in VLMs for 3D Indoor Scene Generation

arXiv 2503.18476 ↗First posted 24 Mar 2025cs.CV cs.CLcited by 29

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Large Vision-Language Models (VLMs), such as GPT-4, have achieved remarkable success across various fields. However, there are few studies on 3D indoor scene generation with VLMs. This paper considers this task as a planning problem subject to spatial and layout common sense constraints. To solve the problem with a VLM, we propose a new global-local tree search algorithm. Globally, the method places each object sequentially and explores multiple placements during each placement process, where the problem space is represented as a tree. To reduce the depth of the tree, we decompose the scene structure hierarchically, i.e. room level, region level, floor object level, and supported object level. The algorithm independently generates the floor objects in different regions and supported objects placed on different floor objects. Locally, we also decompose the sub-task, the placement of each object, into multiple steps. The algorithm searches the tree of problem space. To leverage the VLM model to produce positions of objects, we discretize the top-down view space as a dense grid and fill each cell with diverse emojis to make to cells distinct. We prompt the VLM with the emoji grid and the VLM produces a reasonable location for the object by describing the position with the name of emojis. The quantitative and qualitative experimental results illustrate our approach generates more plausible 3D scenes than state-of-the-art approaches. Our source code is available at https://github.com/dw-dengwei/TreeSearchGen .

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 35 of its 38 references that are in the map. First posted on arXiv 24 Mar 2025.

Builds on

35 of 38 references.

References · 35

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  8. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  9. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
  10. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  11. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,540
  12. 2020GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fieldscited by 785
23 more
  1. 2023Objaverse-XL: A Universe of 10M+ 3D Objectscited by 774
  2. 2023Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creationcited by 687
  3. 2022Towards Reasoning in Large Language Models: A Surveycited by 596
  4. 20203D-FRONT: 3D Furnished Rooms with layOuts and semaNTicscited by 380
  5. 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 347
  6. 20203D-FUTURE: 3D Furniture shape with TextUREcited by 311
  7. 2023Holodeck: Language Guided Generation of 3D Embodied AI Environmentscited by 268
  8. 2020Learning 3D Semantic Scene Graphs from 3D Indoor Reconstructionscited by 208
  9. 2021ATISS: Autoregressive Transformers for Indoor Scene Synthesiscited by 200
  10. 2020BlockGAN: Learning 3D Object-aware Scene Representations from Unlabelled Imagescited by 166
  11. 2023DiffuScene: Denoising Diffusion Models for Generative Indoor Scene Synthesiscited by 166
  12. 2020SceneFormer: Indoor Scene Generation with Transformerscited by 158
  13. 2024PhyScene: Physically Interactable 3D Scene Synthesis for Embodied AIcited by 125
  14. 2024InstructScene: Instruction-Driven 3D Indoor Scene Synthesis with Semantic Graph Priorcited by 117
  15. 2024GALA3D: Towards Text-to-3D Complex Scene Generation via Layout-guided Generative Gaussian Splattingcited by 109
  16. 2021Graph-to-3D: End-to-End Generation and Manipulation of 3D Scenes Using Scene Graphscited by 84
  17. 2023AnyHome: Open-Vocabulary Generation of Structured and Textured 3D Homescited by 82
  18. 2018Attentive Relational Networks for Mapping Images to Scene Graphscited by 73
  19. 2023CommonScenes: Generating Commonsense 3D Indoor Scenes with Scene Graph Diffusioncited by 54
  20. 2023SceneHGN: Hierarchical Graph Networks for 3D Indoor Scene Generation with Fine-Grained Geometrycited by 39
  21. 2023SGFormer: Semantic Graph Transformer for Point Cloud-based 3D Scene Graph Generationcited by 27
  22. 2023Disentangled Counterfactual Learning for Physical Audiovisual Commonsense Reasoningcited by 13
  23. 2023RDFC-GAN: RGB-Depth Fusion CycleGAN for Indoor Depth Completioncited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 12; 2026: 17.

29 papers.

Sort:

By area:

  1. 2025FlashWorld: High-quality 3D Scene Generation within Secondscited by 32
  2. 2025Direct Numerical Layout Generation for 3D Indoor Scene Synthesis via Spatial Reasoningcited by 28
  3. 2025HOLODECK 2.0: Vision-Language-Guided 3D World Generation with Editingcited by 15
  4. 2025Video Perception Models for 3D Scene Synthesiscited by 7
  5. 2025DisCo-Layout: Disentangling and Coordinating Semantic and Physical Refinement in a Multi-Agent Framework for 3D Indoor Layout Synthesiscited by 6
  6. 2026SceneAssistant: A Visual Feedback Agent for Open-Vocabulary 3D Scene Generationcited by 6
  7. 2026FlowScene: Style-Consistent Indoor Scene Generation with Multimodal Graph Rectified Flowcited by 5
  8. 2025Robo-SGG: Exploiting Layout-Oriented Normalization and Restitution Can Improve Robust Scene Graph Generationcited by 3
  9. 2026Edit-As-Act: Goal-Regressive Planning for Open-Vocabulary 3D Indoor Scene Editingcited by 3
  10. 2026Active Exploring like a Pigeon: Reinforcing Spatial Reasoning via Agentic Vision-Language Modelscited by 3
  11. 20253DResT: A Strong Baseline for Semi-Supervised 3D Referring Expression Segmentationcited by 2
  12. 2025Investigating Domain Gaps for Indoor 3D Object Detectioncited by 2
17 more
  1. 2026SpatialGrammar: A Domain-Specific Language for LLM-Based 3D Indoor Scene Generationcited by 2
  2. 2025Chain-of-Evidence Multimodal Reasoning for Few-shot Temporal Action Localizationcited by 1
  3. 2025SceneLCM: End-to-End Layout-Guided Interactive Indoor Scene Generation with Latent Consistency Modelcited by 1
  4. 2025Apples on the Table? Evaluating Text-Guided 3D Scene Synthesis via Fine-Grained Constraint Verificationcited by 1
  5. 2026HOG-Layout: Hierarchical 3D Scene Generation, Optimization and Editing via Vision-Language Modelscited by 1
  6. 2026Pair2Scene: Learning Local Object Relations for Procedural Scene Generationcited by 1
  7. 2026CasLayout: Cascaded 3D Layout Diffusion for Indoor Scene Synthesis with Implicit Relation Modelingcited by 1
  8. 2026Function2Scene: 3D Indoor Scene Layout from Functional Specificationscited by 1
  9. 2026Global-Local Monte Carlo Tree Search in Vision-Language Models for Text-to-3D Indoor Scene Generationcited by 1
  10. 2026iARCS: Iterative Agentic RL for Controllable 3D Scene Generationcited by 1
  11. 2025Multi-Modal Scene Graph with Kolmogorov-Arnold Experts for Audio-Visual Question Answeringcited by 0
  12. 2026SDesc3D: Towards Layout-Aware 3D Indoor Scene Generation from Short Descriptionscited by 0
  13. 2026SGFormer++: Semantic Graph Transformer for Incremental 3D Scene Graph Generationcited by 0
  14. 2026VTO: Visual Tool Orchestration for Video Anomaly Detectioncited by 0
  15. 2026PolyLayout: Hierarchical VLM-Guided Layout Generation Beyond Rectangular Roomscited by 0
  16. 2026SceneMosaic: Efficient and Diverse Simulation-Ready Scene Generation via Hybrid Agentic Layout Evolutioncited by 0
  17. 2026Frozen Scenes, Shifting Winners: Configuration Fragility in Text-to-3D Evaluationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Global Graph-Validated Optimization for VLM-based 3D Indoor Scene Generation
  2. 2024ROOT: VLM based System for Indoor Scene Understanding and Beyond
  3. 2024Open-Universe Indoor Scene Generation using LLM Program Synthesis and Uncurated Object Databases
  4. 2026ThinkBLOX: 3D Indoor Scene Generation with Progressive Reasoning
  5. 2025SpatialLM: Training Large Language Models for Structured Indoor Modeling
  6. 2024VLA-3D: A Dataset for 3D Semantic Scene Understanding and Navigation
  7. 2026HDSL: A Hierarchical Domain-Specific Language for Structured 3D Indoor Scene Generation and Localized Editing with LLM Agents
  8. 2025Decorum: A Language-Based Approach For Style-Conditioned Synthesis of Indoor 3D Scenes