# Text to 3D Scene Generation with Rich Lexical Grounding

arXiv: https://arxiv.org/abs/1505.06289 · HTML: https://mlsanity.com/paper/1505.06289/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 May 2015.

## Built on by (most cited first, 30 of 36)

- [Zero-Shot Text-Guided Object Generation with Dream Fields](https://mlsanity.com/paper/2112.01455/) (2021)
- [ProcTHOR: Large-Scale Embodied AI Using Procedural Generation](https://mlsanity.com/paper/2206.06994/) (2022)
- [SceneCraft: An LLM Agent for Synthesizing 3D Scene as Blender Code](https://mlsanity.com/paper/2403.01248/) (2024)
- [InstructScene: Instruction-Driven 3D Indoor Scene Synthesis with Semantic Graph Prior](https://mlsanity.com/paper/2402.04717/) (2024)
- [GALA3D: Towards Text-to-3D Complex Scene Generation via Layout-guided Generative Gaussian Splatting](https://mlsanity.com/paper/2402.07207/) (2024)
- [RealmDreamer: Text-Driven 3D Scene Generation with Inpainting and Depth Diffusion](https://mlsanity.com/paper/2404.07199/) (2024)
- [I-Design: Personalized LLM Interior Designer](https://mlsanity.com/paper/2404.02838/) (2024)
- [EchoScene: Indoor Scene Generation via Information Echo over Scene Graph Diffusion](https://mlsanity.com/paper/2405.00915/) (2024)
- [Graphologue: Exploring Large Language Model Responses with Interactive Diagrams](https://mlsanity.com/paper/2305.11473/) (2023)
- [MIME: Human-Aware 3D Scene Generation](https://mlsanity.com/paper/2212.04360/) (2022)
- [Grounding 'Grounding' in NLP](https://mlsanity.com/paper/2106.02192/) (2021)
- [SceneHGN: Hierarchical Graph Networks for 3D Indoor Scene Generation with Fine-Grained Geometry](https://mlsanity.com/paper/2302.10237/) (2023)
- [Indoor Scene Generation from a Collection of Semantic-Segmented Depth Images](https://mlsanity.com/paper/2108.09022/) (2021)
- [SceneEval: Evaluating Semantic Coherence in Text-Conditioned 3D Indoor Scene Synthesis](https://mlsanity.com/paper/2503.14756/) (2025)
- [A Theory of Multimodal Learning](https://mlsanity.com/paper/2309.12458/) (2023)
- [Chat2Layout: Interactive 3D Furniture Layout with a Multimodal LLM](https://mlsanity.com/paper/2407.21333/) (2024)
- [Neural MP: A Generalist Neural Motion Planner](https://mlsanity.com/paper/2409.05864/) (2024)
- [XVO: Generalized Visual Odometry via Cross-Modal Self-Training](https://mlsanity.com/paper/2309.16772/) (2023)
- [OptiScene: LLM-driven Indoor Scene Layout Generation via Scaled Human-aligned Data Synthesis and Multi-Stage Preference Optimization](https://mlsanity.com/paper/2506.07570/) (2025)
- [Aladdin: Zero-Shot Hallucination of Stylized 3D Assets from Abstract Scene Descriptions](https://mlsanity.com/paper/2306.06212/) (2023)
- [Language-driven Scene Synthesis using Multi-conditional Diffusion Model](https://mlsanity.com/paper/2310.15948/) (2023)
- [PlaceIt3D: Language-Guided Object Placement in Real 3D Scenes](https://mlsanity.com/paper/2505.05288/) (2025)
- [FreeScene: Mixed Graph Diffusion for 3D Scene Synthesis from Free Prompts](https://mlsanity.com/paper/2506.02781/) (2025)
- [GS-Agent: Creating 4D Physical Worlds With Generative Simulation](https://mlsanity.com/paper/2607.21522/) (2026)
- [Point Cloud Generation with Continuous Conditioning](https://mlsanity.com/paper/2202.08526/) (2022)
- [Multimodal Classification via Modal-Aware Interactive Enhancement](https://mlsanity.com/paper/2407.04587/) (2024)
- [Automatic Scene Generation: State-of-the-Art Techniques, Models, Datasets, Challenges, and Future Prospects](https://mlsanity.com/paper/2410.01816/) (2024)
- [DreamPartGen: Semantically Grounded Part-Level 3D Generation via Collaborative Latent Denoising](https://mlsanity.com/paper/2603.19216/) (2026)
- [On the Computational Benefit of Multimodal Learning](https://mlsanity.com/paper/2309.13782/) (2023)
- [HOG-Layout: Hierarchical 3D Scene Generation, Optimization and Editing via Vision-Language Models](https://mlsanity.com/paper/2604.10772/) (2026)

## Abstract (arXiv)

The ability to map descriptions of scenes to 3D geometric representations has many applications in areas such as art, education, and robotics. However, prior work on the text to 3D scene generation task has used manually specified object categories and language that identifies them. We introduce a dataset of 3D scenes annotated with natural language descriptions and learn from this data how to ground textual descriptions to physical objects. Our method successfully grounds a variety of lexical terms to concrete referents, and we show quantitatively that our method improves 3D scene generation over previous work using purely rule-based methods. We evaluate the fidelity and plausibility of 3D scenes generated with our grounding approach through human judgments. To ease evaluation on this task, we also introduce an automated metric that strongly correlates with human judgments.