# SceneSeer: 3D Scene Design with Natural Language

arXiv: https://arxiv.org/abs/1703.00050 · HTML: https://mlsanity.com/paper/1703.00050/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Feb 2017.

## Built on by (most cited first, 21 of 21)

- [Holodeck: Language Guided Generation of 3D Embodied AI Environments](https://mlsanity.com/paper/2312.09067/) (2023)
- [DiffuScene: Denoising Diffusion Models for Generative Indoor Scene Synthesis](https://mlsanity.com/paper/2303.14207/) (2023)
- [SceneCraft: An LLM Agent for Synthesizing 3D Scene as Blender Code](https://mlsanity.com/paper/2403.01248/) (2024)
- [InstructScene: Instruction-Driven 3D Indoor Scene Synthesis with Semantic Graph Prior](https://mlsanity.com/paper/2402.04717/) (2024)
- [AnyHome: Open-Vocabulary Generation of Structured and Textured 3D Homes](https://mlsanity.com/paper/2312.06644/) (2023)
- [MIME: Human-Aware 3D Scene Generation](https://mlsanity.com/paper/2212.04360/) (2022)
- [3DALL-E: Integrating Text-to-Image AI in 3D Design Workflows](https://mlsanity.com/paper/2210.11603/) (2022)
- [CityCraft: A Real Crafter for 3D City Generation](https://mlsanity.com/paper/2406.04983/) (2024)
- [Learning 3D Scene Priors with 2D Supervision](https://mlsanity.com/paper/2211.14157/) (2022)
- [SceneGenAgent: Precise Industrial Scene Generation with Coding Agent](https://mlsanity.com/paper/2410.21909/) (2024)
- [RoomDesigner: Encoding Anchor-latents for Style-consistent and Shape-compatible Indoor Scene Generation](https://mlsanity.com/paper/2310.10027/) (2023)
- [Language-driven Scene Synthesis using Multi-conditional Diffusion Model](https://mlsanity.com/paper/2310.15948/) (2023)
- [PlaceIt3D: Language-Guided Object Placement in Real 3D Scenes](https://mlsanity.com/paper/2505.05288/) (2025)
- [FreeScene: Mixed Graph Diffusion for 3D Scene Synthesis from Free Prompts](https://mlsanity.com/paper/2506.02781/) (2025)
- [From Programs to Poses: Factored Real-World Scene Generation via Learned Program Libraries](https://mlsanity.com/paper/2510.10292/) (2025)
- [S-INF: Towards Realistic Indoor Scene Synthesis via Scene Implicit Neural Field](https://mlsanity.com/paper/2412.17561/) (2024)
- [Computer-Aided Layout Generation for Building Design: A Review](https://mlsanity.com/paper/2504.09694/) (2025)
- [3D-Generalist: Self-Improving Vision-Language-Action Models for Crafting 3D Worlds](https://mlsanity.com/paper/2507.06484/) (2025)
- [HOG-Layout: Hierarchical 3D Scene Generation, Optimization and Editing via Vision-Language Models](https://mlsanity.com/paper/2604.10772/) (2026)
- [Global Graph-Validated Optimization for VLM-based 3D Indoor Scene Generation](https://mlsanity.com/paper/2608.03064/) (2026)
- [SPHERE: Adaptive VR Indoor Scene Generation via LLM-Enhanced Spatial Preference Learning and Human-in-the-Loop RL](https://mlsanity.com/paper/2610.02023/) (2026)

## Abstract (arXiv)

Designing 3D scenes is currently a creative task that requires significant expertise and effort in using complex 3D design interfaces. This effortful design process starts in stark contrast to the easiness with which people can use language to describe real and imaginary environments. We present SceneSeer: an interactive text to 3D scene generation system that allows a user to design 3D scenes using natural language. A user provides input text from which we extract explicit constraints on the objects that should appear in the scene. Given these explicit constraints, the system then uses a spatial knowledge base learned from an existing database of 3D scenes and 3D object models to infer an arrangement of the objects forming a natural scene matching the input description. Using textual commands the user can then iteratively refine the created scene by adding, removing, replacing, and manipulating objects. We evaluate the quality of 3D scenes generated by SceneSeer in a perceptual evaluation experiment where we compare against manually designed scenes and simpler baselines for 3D scene generation. We demonstrate how the generated scenes can be iteratively refined through simple natural language commands.