mlsanityData to 7 Oct 2026

InstructScene: Instruction-Driven 3D Indoor Scene Synthesis with Semantic Graph Prior

arXiv 2402.04717 ↗First posted 7 Feb 2024cs.CVcited by 110

Not on the map. Why?

Abstract (arXiv)

Comprehending natural language instructions is a charming property for 3D indoor scene synthesis systems. Existing methods directly model object joint distributions and express object relations implicitly within a scene, thereby hindering the controllability of generation. We introduce InstructScene, a novel generative framework that integrates a semantic graph prior and a layout decoder to improve controllability and fidelity for 3D scene synthesis. The proposed semantic graph prior jointly learns scene appearances and layout distributions, exhibiting versatility across various downstream tasks in a zero-shot manner. To facilitate the benchmarking for text-driven 3D scene synthesis, we curate a high-quality dataset of scene-instruction pairs with large language and multimodal models. Extensive experimental results reveal that the proposed method surpasses existing state-of-the-art approaches by a large margin. Thorough ablation studies confirm the efficacy of crucial design components. Project page: https://chenguolin.github.io/projects/InstructScene.

Key facts. Cited by 110 papers in the map (data through 7 Oct 2026). Builds on 63 of its 83 references that are in the map. First posted on arXiv 7 Feb 2024.

Builds on

63 of 83 references.

References · 63

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  7. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  9. 2022Classifier-Free Diffusion Guidancecited by 5,717
  10. 2016Layer Normalizationcited by 5,579
  11. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  12. 2021Zero-Shot Text-to-Image Generationcited by 4,276
38 more
  1. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  2. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  3. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  4. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  5. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  6. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  7. 2018Demystifying MMD GANscited by 1,268
  8. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,186
  9. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  10. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
  11. 2023Objaverse-XL: A Universe of 10M+ 3D Objectscited by 767
  12. 2020A Generalization of Transformer Networks to Graphscited by 653
  13. 2021Argmax Flows and Multinomial Diffusion: Learning Categorical Distributionscited by 566
  14. 2022DiGress: Discrete Denoising diffusion for graph generationcited by 503
  15. 2018GraphRNN: Generating Realistic Graphs with Deep Auto-regressive Modelscited by 406
  16. 2018MolGAN: An implicit generative model for small molecular graphscited by 392
  17. 2018GraphVAE: Towards Generation of Small Graphs Using Variational Autoencoderscited by 375
  18. 20203D-FRONT: 3D Furnished Rooms with layOuts and semaNTicscited by 375
  19. 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 337
  20. 2018Image Generation from Scene Graphscited by 324
  21. 20203D-FUTURE: 3D Furniture shape with TextUREcited by 306
  22. 2022Score-based Generative Modeling of Graphs via the System of Stochastic Differential Equationscited by 273
  23. 2020Permutation Invariant Graph Generation via Score-Based Generative Modelingcited by 263
  24. 2023Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Modelscited by 245
  25. 2018Learning Deep Generative Models of Graphscited by 227
  26. 2022The Role of ImageNet Classes in Fr'echet Inception Distancecited by 211
  27. 2021ATISS: Autoregressive Transformers for Indoor Scene Synthesiscited by 195
  28. 2023OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understandingcited by 183
  29. 2020SceneFormer: Indoor Scene Generation with Transformerscited by 154
  30. 2018Fast and Flexible Indoor Scene Synthesis via Deep Convolutional Generative Modelscited by 105
  31. 2022SPECTRE: Spectral Conditioning Helps to Overcome the Expressivity Limits of One-shot Graph Generatorscited by 96
  32. 2021Graph-to-3D: End-to-End Generation and Manipulation of 3D Scenes Using Scene Graphscited by 83
  33. 2023Autoregressive Diffusion Model for Graph Generationcited by 82
  34. 2023Set-the-Scene: Global-Local Training for Generating Controllable NeRF Scenescited by 75
  35. 2018Human-centric Indoor Scene Synthesis Using Stochastic Grammarcited by 73
  36. 2018Deep Generative Modeling for Scene Synthesis via Hybrid Representationscited by 63
  37. 2023CompoNeRF: Text-guided Multi-object Compositional NeRF with Editable 3D Scene Layoutcited by 57
  38. 2019SceneGraphNet: Neural Message Passing for 3D Indoor Scene Augmentationcited by 55

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 17; 2025: 39; 2026: 54.

110 papers.

Sort:

By area:

  1. 2024LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Modelscited by 94
  2. 2024I-Design: Personalized LLM Interior Designercited by 87
  3. 2025PartCrafter: Structured 3D Mesh Generation via Compositional Latent Diffusion Transformerscited by 78
  4. 2024EchoScene: Indoor Scene Generation via Information Echo over Scene Graph Diffusioncited by 66
  5. 2024SINGAPO: Single Image Controlled Generation of Articulated Parts in Objectscited by 66
  6. 2025Scenethesis: A Language and Vision Agentic Framework for 3D Scene Generationcited by 65
  7. 2024LLplace: The 3D Indoor Scene Layout Generation and Editing via Large Language Modelcited by 34
  8. 2024Mixed Diffusion for 3D Indoor Scene Synthesiscited by 30
  9. 2025Global-Local Tree Search in VLMs for 3D Indoor Scene Generationcited by 29
  10. 2025ArtiScene: Language-Driven Artistic 3D Scene Generation Through Image Intermediarycited by 27
  11. 2025Direct Numerical Layout Generation for 3D Indoor Scene Synthesis via Spatial Reasoningcited by 27
  12. 2025SceneEval: Evaluating Semantic Coherence in Text-Conditioned 3D Indoor Scene Synthesiscited by 26
38 more
  1. 2025HSM: Hierarchical Scene Motifs for Multi-Scale Indoor Scene Generationcited by 22
  2. 2024Chat2Layout: Interactive 3D Furniture Layout with a Multimodal LLMcited by 20
  3. 2024DeBaRA: Denoising-Based 3D Room Arrangement Generationcited by 20
  4. 2025MMGDreamer: Mixed-Modality Graph for Geometry-Controllable 3D Indoor Scene Generationcited by 18
  5. 2025HiScene: Creating Hierarchical 3D Scenes with Isometric View Generationcited by 18
  6. 2025WorldGen: From Text to Traversable and Interactive 3D Worldscited by 18
  7. 2024EditRoom: LLM-parameterized Graph Diffusion for Composable 3D Room Layout Editingcited by 14
  8. 2025CasaGPT: Cuboid Arrangement and Scene Assembly for Interior Designcited by 14
  9. 2025OptiScene: LLM-driven Indoor Scene Layout Generation via Scaled Human-aligned Data Synthesis and Multi-Stage Preference Optimizationcited by 14
  10. 2024Diorama: Unleashing Zero-shot Single-view 3D Indoor Scene Modelingcited by 10
  11. 2024Planner3D: LLM-enhanced graph prior meets 3D indoor scene explicit regularizationcited by 9
  12. 2025Controllable 3D Outdoor Scene Generation via Scene Graphscited by 9
  13. 2025CHOrD: Generation of Collision-Free, House-Scale, and Organized Digital Twins for 3D Indoor Scenes with Controllable Floor Plans and Optimal Layoutscited by 9
  14. 2025RoomCraft: Controllable and Complete 3D Indoor Scene Generationcited by 9
  15. 2025FreeScene: Mixed Graph Diffusion for 3D Scene Synthesis from Free Promptscited by 8
  16. 2024DreamScape: 3D Scene Creation via Gaussian Splatting joint Correlation Modelingcited by 7
  17. 2024Lay-A-Scene: Personalized 3D Object Arrangement Using Text-to-Image Priorscited by 7
  18. 2024TextLap: Customizing Language Models for Text-to-Layout Planningcited by 7
  19. 2025Digital Twin Generation from Visual Data: A Surveycited by 7
  20. 2025Video Perception Models for 3D Scene Synthesiscited by 7
  21. 2025Diff4Splat: Controllable 4D Scene Generation with Latent Dynamic Reconstruction Modelscited by 7
  22. 2026SceneCode: Executable World Programs for Editable Indoor Scenes with Articulated Objectscited by 7
  23. 2024InstructLayout: Instruction-Driven 2D and 3D Layout Synthesis with Semantic Graph Priorcited by 6
  24. 2025SemLayoutDiff: Semantic Layout Generation with Diffusion Model for Indoor Scene Synthesiscited by 6
  25. 2026SceneReVis: A Self-Reflective Vision-Grounded Framework for 3D Indoor Scene Synthesis via Multi-turn RLcited by 6
  26. 2026REST3D: Reconstructing Physically Stable 3D Scenes from a Single Imagecited by 6
  27. 2025ReSpace: Text-Driven Autoregressive 3D Indoor Scene Synthesis and Editingcited by 5
  28. 2025ImmerseGen: Agent-Guided Immersive World Generation with Alpha-Textured Proxiescited by 5
  29. 2025M3DLayout: A Multi-Source Dataset of 3D Indoor Layouts and Structured Descriptions for 3D Generationcited by 5
  30. 2025MoCA: Mixture-of-Components Attention for Scalable Compositional 3D Generationcited by 5
  31. 2026AnchoredDream: Zero-Shot 360deg Indoor Scene Generation from a Single View via Geometric Groundingcited by 5
  32. 2025ScanEdit: Hierarchically-Guided Functional 3D Scan Editingcited by 4
  33. 2025HLG: Comprehensive 3D Room Construction via Hierarchical Layout Generationcited by 4
  34. 2026FlowScene: Style-Consistent Indoor Scene Generation with Multimodal Graph Rectified Flowcited by 4
  35. 2026SimWorlds: A Multi-Agent System for Dynamic 3D Scene Creationcited by 4
  36. 2024S-INF: Towards Realistic Indoor Scene Synthesis via Scene Implicit Neural Fieldcited by 3
  37. 20253D-Generalist: Self-Improving Vision-Language-Action Models for Crafting 3D Worldscited by 3
  38. 20253D Software Synthesis Driven by Constraint-Expressive Intermediate Representationcited by 3

50 most cited of 110 All 110 (JSON)