mlsanityData to 7 Oct 2026

SceneTeller: Language-to-3D Scene Generation

arXiv 2407.20727 ↗First posted 30 Jul 2024cs.CVcited by 43

Not on the map. Why?

Abstract (arXiv)

Designing high-quality indoor 3D scenes is important in many practical applications, such as room planning or game development. Conventionally, this has been a time-consuming process which requires both artistic skill and familiarity with professional software, making it hardly accessible for layman users. However, recent advances in generative AI have established solid foundation for democratizing 3D design. In this paper, we propose a pioneering approach for text-based 3D room design. Given a prompt in natural language describing the object placement in the room, our method produces a high-quality 3D scene corresponding to it. With an additional text prompt the users can change the appearance of the entire scene or of individual objects in it. Built using in-context learning, CAD model retrieval and 3D-Gaussian-Splatting-based stylization, our turnkey pipeline produces state-of-the-art 3D scenes, while being easy to use even for novices. Our project page is available at https://sceneteller.github.io/.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). Builds on 53 of its 57 references that are in the map. First posted on arXiv 30 Jul 2024.

Builds on

53 of 57 references.

References · 53

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  7. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  8. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  9. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  10. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  11. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  12. 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,957
38 more
  1. 2021Plenoxels: Radiance Fields without Neural Networkscited by 1,607
  2. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  3. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,285
  4. 2022TensoRF: Tensorial Radiance Fieldscited by 1,275
  5. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillationcited by 1,207
  6. 2021Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstructioncited by 961
  7. 2021PlenOctrees for Real-time Rendering of Neural Radiance Fieldscited by 904
  8. 2023DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creationcited by 837
  9. 2022Block-NeRF: Scalable Large Scene Neural View Synthesiscited by 720
  10. 2023Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creationcited by 684
  11. 2022Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generationcited by 597
  12. 2023Nerfstudio: A Modular Framework for Neural Radiance Field Developmentcited by 596
  13. 2022Latent-NeRF for Shape-Guided Generation of 3D Shapes and Texturescited by 506
  14. 2023Instruct-NeRF2NeRF: Editing 3D Scenes with Instructionscited by 482
  15. 2023Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priorscited by 395
  16. 20203D-FRONT: 3D Furnished Rooms with layOuts and semaNTicscited by 375
  17. 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 337
  18. 2023GaussianEditor: Swift and Controllable 3D Editing with Gaussian Splattingcited by 331
  19. 20203D-FUTURE: 3D Furniture shape with TextUREcited by 306
  20. 2023Text-to-3D using Gaussian Splattingcited by 299
  21. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 298
  22. 2023LucidDreamer: Towards High-Fidelity Text-to-3D Generation via Interval Score Matchingcited by 290
  23. 2023Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Modelscited by 245
  24. 2023DreamBooth3D: Subject-Driven Text-to-3D Generationcited by 235
  25. 2023LucidDreamer: Domain-free Generation of 3D Gaussian Splatting Scenescited by 228
  26. 2023MERF: Memory-Efficient Radiance Fields for Real-time View Synthesis in Unbounded Scenescited by 215
  27. 2021ATISS: Autoregressive Transformers for Indoor Scene Synthesiscited by 195
  28. 2023GaussianEditor: Editing 3D Gaussians Delicately with Text Instructionscited by 166
  29. 2023Grounded Text-to-Image Synthesis with Attention Refocusingcited by 163
  30. 2023SceneScape: Text-Driven Consistent Scene Generationcited by 160
  31. 2021Hallucinated Neural Radiance Fields in the Wildcited by 130
  32. 2023HiFA: High-fidelity Text-to-3D Generation with Advanced Diffusion Guidancecited by 109
  33. 2024GALA3D: Towards Text-to-3D Complex Scene Generation via Layout-guided Generative Gaussian Splattingcited by 104
  34. 2023Compositional 3D Scene Generation using Locally Conditioned Diffusioncited by 101
  35. 20233D-GPT: Procedural 3D Modeling with Large Language Modelscited by 91
  36. 2023ATT3D: Amortized Text-to-3D Object Synthesiscited by 87
  37. 2023Set-the-Scene: Global-Local Training for Generating Controllable NeRF Scenescited by 75
  38. 2022DisCoScene: Spatially Disentangled Generative Radiance Fields for Controllable 3D-aware Scene Synthesiscited by 61

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 25; 2026: 17.

43 papers.

Sort:

By area:

  1. 2025Scenethesis: A Language and Vision Agentic Framework for 3D Scene Generationcited by 65
  2. 2025VLM4D: Towards Spatiotemporal Awareness in Vision Language Modelscited by 43
  3. 2025ArtiScene: Language-Driven Artistic 3D Scene Generation Through Image Intermediarycited by 27
  4. 20253D Scene Generation: A Surveycited by 24
  5. 2025HiScene: Creating Hierarchical 3D Scenes with Isometric View Generationcited by 18
  6. 2025OptiScene: LLM-driven Indoor Scene Layout Generation via Scaled Human-aligned Data Synthesis and Multi-Stage Preference Optimizationcited by 14
  7. 2025HOLODECK 2.0: Vision-Language-Guided 3D World Generation with Editingcited by 14
  8. 2024Articulate3D: Holistic Understanding of 3D Scenes as Universal Scene Descriptioncited by 9
  9. 2025Towards Geometric and Textural Consistency 3D Scene Generation via Single Image-guided Model Generation and Layout Optimizationcited by 6
  10. 2025Learning 3D Object Spatial Relationships from Pre-trained 2D Diffusion Modelscited by 5
  11. 2025ImmerseGen: Agent-Guided Immersive World Generation with Alpha-Textured Proxiescited by 5
  12. 2026SceneAssistant: A Visual Feedback Agent for Open-Vocabulary 3D Scene Generationcited by 5
31 more
  1. 2025Learning 3D Scene Analogies with Neural Contextual Scene Mapscited by 4
  2. 2025ScanEdit: Hierarchically-Guided Functional 3D Scan Editingcited by 4
  3. 2025From 2D to 3D Cognition: A Brief Survey of General World Modelscited by 4
  4. 2025HLG: Comprehensive 3D Room Construction via Hierarchical Layout Generationcited by 4
  5. 2026Imagine a City: CityGenAgent for Procedural 3D City Generationcited by 4
  6. 2026FlowScene: Style-Consistent Indoor Scene Generation with Multimodal Graph Rectified Flowcited by 4
  7. 2026SimWorlds: A Multi-Agent System for Dynamic 3D Scene Creationcited by 4
  8. 20253D Software Synthesis Driven by Constraint-Expressive Intermediate Representationcited by 3
  9. 2026Vinedresser3D: Agentic Text-guided 3D Editingcited by 3
  10. 2026StoryBlender: Inter-Shot Consistent and Editable 3D Storyboard with Spatial-temporal Dynamicscited by 3
  11. 2025TabletopGen: Tabletop Scene Generation and Interactive Simulation for Robotic Manipulationcited by 2
  12. 2026SpatialGrammar: A Domain-Specific Language for LLM-Based 3D Indoor Scene Generationcited by 2
  13. 2026NaLA: A 3D Native LLM Layout Agent for High-quality 3D Scene Generationcited by 2
  14. 2025AccioScene: Compositional 3D Scene Generation via Graph Diffusion and Interaction-driven Criticscited by 1
  15. 2025From Fields to Splats: A Cross-Domain Survey of Real-Time Neural Scene Representationscited by 1
  16. 2025HiGS: Hierarchical Generative Scene Framework for Multi-Step Associative Semantic Spatial Compositioncited by 1
  17. 2025Apples on the Table? Evaluating Text-Guided 3D Scene Synthesis via Fine-Grained Constraint Verificationcited by 1
  18. 2025RoomPlanner: Explicit Layout Planner for Easier LLM-Driven 3D Room Generationcited by 1
  19. 2026Copy-Trasform-Paste: Zero-Shot Object-Object Alignment Guided by Vision-Language and Geometric Constraintscited by 1
  20. 2026CasLayout: Cascaded 3D Layout Diffusion for Indoor Scene Synthesis with Implicit Relation Modelingcited by 1
  21. 2026OrbiSim: World Models as Differentiable Physics Engines for Embodied Intelligencecited by 1
  22. 2025FurniMAS: Language-Guided Furniture Decoration using Multi-Agent Systemcited by 0
  23. 2025ASSIST-3D: Adapted Scene Synthesis for Class-Agnostic 3D Instance Segmentationcited by 0
  24. 2025RoomPilot: Controllable Indoor Scene Synthesis via Multimodal Semantic Parsingcited by 0
  25. 2025Sceniris: A Fast Procedural Scene Generation Frameworkcited by 0
  26. 2026Text-to-Stage: Spatial Layouts from Long-form Narrativescited by 0
  27. 2026Function2Scene: 3D Indoor Scene Layout from Functional Specificationscited by 0
  28. 2026Interpretable Human-Label-Free Deep Learning for Real-Bogus Classification with Uncertainty Quantificationcited by 0
  29. 2026ThinkBLOX: 3D Indoor Scene Generation with Progressive Reasoningcited by 0
  30. 2026aDSL: Agentic 3D Creation via Joint Agent-Program Designcited by 0
  31. 2026ScenePilot: Grow-and-Repair Policy for Text-Driven 3D Indoor Scene Generationcited by 0