mlsanityData to 7 Oct 2026

Holodeck: Language Guided Generation of 3D Embodied AI Environments

arXiv 2312.09067 ↗First posted 14 Dec 2023cs.CV cs.AI cs.CLcited by 257

On the map: World models → Video-generation world models and simulators

Abstract (arXiv)

3D simulated environments play a critical role in Embodied AI, but their creation requires expertise and extensive manual effort, restricting their diversity and scope. To mitigate this limitation, we present Holodeck, a system that generates 3D environments to match a user-supplied prompt fully automatedly. Holodeck can generate diverse scenes, e.g., arcades, spas, and museums, adjust the designs for styles, and can capture the semantics of complex queries such as "apartment for a researcher with a cat" and "office of a professor who is a fan of Star Wars". Holodeck leverages a large language model (i.e., GPT-4) for common sense knowledge about what the scene might look like and uses a large collection of 3D assets from Objaverse to populate the scene with diverse objects. To address the challenge of positioning objects correctly, we prompt GPT-4 to generate spatial relational constraints between objects and then optimize the layout to satisfy those constraints. Our large-scale human evaluation shows that annotators prefer Holodeck over manually designed procedural baselines in residential scenes and that Holodeck can produce high-quality outputs for diverse scene types. We also demonstrate an exciting application of Holodeck in Embodied AI, training agents to navigate in novel scenes like music rooms and daycares without human-constructed data, which is a significant step forward in developing general-purpose embodied agents.

Key facts. Cited by 257 papers in the map (data through 7 Oct 2026). Placed in Video-generation world models and simulators, one of 11 directions in World models. Builds on 43 of its 58 references that are in the map. First posted on arXiv 14 Dec 2023.

Builds on

43 of 58 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022DreamFusion: Text-to-3D using 2D Diffusion29 Sep
  2. 2021Learning Transferable Visual Models From Natural Language Supervision26 Feb
  3. 2020ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation9 Jul
  4. 2017AI2-THOR: An Interactive 3D Environment for Visual AI14 Dec

Other references · 39

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  3. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  4. 2022Emergent Abilities of Large Language Modelscited by 2,483
  5. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  6. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  7. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,285
  8. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  9. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 579
  10. 2016Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modelingcited by 558
  11. 20213D Shape Generation and Completion through Point-Voxel Diffusioncited by 516
  12. 2022Latent-NeRF for Shape-Guided Generation of 3D Shapes and Texturescited by 506
27 more
  1. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 501
  2. 2019PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flowscited by 487
  3. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 477
  4. 2018VirtualHome: Simulating Household Activities via Programscited by 411
  5. 20203D-FRONT: 3D Furnished Rooms with layOuts and semaNTicscited by 375
  6. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 369
  7. 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 337
  8. 2023Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Modelscited by 245
  9. 2020ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objectscited by 240
  10. 2023RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulationcited by 206
  11. 2021ATISS: Autoregressive Transformers for Indoor Scene Synthesiscited by 195
  12. 2021Simple but Effective: CLIP Embeddings for Embodied AIcited by 191
  13. 2020RoboTHOR: An Open Simulation-to-Real Embodied AI Platformcited by 190
  14. 2023NerfDiff: Single-image View Synthesis with NeRF-guided Distillation from 3D-aware Diffusioncited by 179
  15. 2018Escaping Plato's Cave: 3D Shape From Adversarial Renderingcited by 170
  16. 2023Habitat Synthetic Scenes Dataset (HSSD-200): An Analysis of 3D Scene Scale and Realism Tradeoffs for ObjectGoal Navigationcited by 162
  17. 2023SceneScape: Text-Driven Consistent Scene Generationcited by 160
  18. 2020SceneFormer: Indoor Scene Generation with Transformerscited by 154
  19. 2023Text2NeRF: Text-Driven 3D Scene Generation with Neural Radiance Fieldscited by 122
  20. 2023LEGO-Net: Learning Regular Rearrangements of Objects in Roomscited by 75
  21. 2022HouseDiffusion: Vector Floorplan Generation via a Diffusion Model with Discrete and Continuous Denoisingcited by 72
  22. 2022Phone2Proc: Bringing Robust Robots Into Our Chaotic Worldcited by 24
  23. 2017SceneSeer: 3D Scene Design with Natural Languagecited by 20
  24. 2023Towards Language-guided Interactive 3D Generation: LLMs as Layout Interpreter with Generative Feedbackcited by 17
  25. 2020Graph2Plan: Learning Floorplan Generation from Layout Graphscited by 15
  26. 2023RoomDesigner: Encoding Anchor-latents for Style-consistent and Shape-compatible Indoor Scene Generationcited by 12
  27. 2023Aladdin: Zero-Shot Hallucination of Stylized 3D Assets from Abstract Scene Descriptionscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 40; 2025: 87; 2026: 130.

257 papers.

Sort:

By area:

  1. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 175
  2. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 161
  3. 2024URDFormer: A Pipeline for Constructing Articulated Simulation Environments from Real-World Imagescited by 106
  4. 2024GALA3D: Towards Text-to-3D Complex Scene Generation via Layout-guided Generative Gaussian Splattingcited by 104
  5. 2024LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Modelscited by 94
  6. 2025Sparc3D: Sparse Representation and Construction for High-Resolution 3D Shapes Modelingcited by 84
  7. 2024AgentGen: Enhancing Planning Abilities for Large Language Model based Agent via Environment and Task Generationcited by 80
  8. 2024Towards Long-Horizon Vision-Language Navigation: Platform, Benchmark and Methodcited by 66
  9. 2025Scenethesis: A Language and Vision Agentic Framework for 3D Scene Generationcited by 65
  10. 20253D and 4D World Modeling: A Surveycited by 64
  11. 2024Human-Object Interaction from Human-Level Instructionscited by 62
  12. 2024PoliFormer: Scaling On-Policy RL with Transformers Results in Masterful Navigatorscited by 55
38 more
  1. 2025EmbodiedEval: Evaluate Multimodal LLMs as Embodied Agentscited by 55
  2. 2025SceneWeaver: All-in-One 3D Scene Synthesis with an Extensible and Self-Reflective Agentcited by 53
  3. 2025SceneGen: Single-Image 3D Scene Generation in One Feedforward Passcited by 49
  4. 2024From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessonscited by 42
  5. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Modelscited by 41
  6. 2025Articulate AnyMesh: Open-Vocabulary 3D Articulated Objects Modelingcited by 41
  7. 2025NavMorph: A Self-Evolving World Model for Vision-and-Language Navigation in Continuous Environmentscited by 41
  8. 2024Architect: Generating Vivid and Interactive 3D Scenes with Hierarchical 2D Inpaintingcited by 37
  9. 20243D-GRAND: A Million-Scale Dataset for 3D-LLMs with Better Grounding and Less Hallucinationcited by 36
  10. 2025Multi-agent Embodied AI: Advances and Future Directionscited by 36
  11. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  12. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  13. 2024LLplace: The 3D Indoor Scene Layout Generation and Editing via Large Language Modelcited by 34
  14. 2024Disentangled 3D Scene Generation with Layout Learningcited by 33
  15. 2025FlashWorld: High-quality 3D Scene Generation within Secondscited by 30
  16. 2025Global-Local Tree Search in VLMs for 3D Indoor Scene Generationcited by 29
  17. 2025Spatial Understanding from Videos: Structured Prompts Meet Simulation Datacited by 28
  18. 2025BlenderGym: Benchmarking Foundational Model Systems for Graphics Editingcited by 27
  19. 2025ArtiScene: Language-Driven Artistic 3D Scene Generation Through Image Intermediarycited by 27
  20. 2025Direct Numerical Layout Generation for 3D Indoor Scene Synthesis via Spatial Reasoningcited by 27
  21. 2025SceneEval: Evaluating Semantic Coherence in Text-Conditioned 3D Indoor Scene Synthesiscited by 26
  22. 2024CityCraft: A Real Crafter for 3D City Generationcited by 25
  23. 2024SceneFactor: Factored Latent 3D Diffusion for Controllable 3D Scene Generationcited by 24
  24. 2025FirePlace: Geometric Refinements of LLM Common Sense Reasoning for 3D Object Placementcited by 24
  25. 20253D Scene Generation: A Surveycited by 24
  26. 2026Gym-Anything: Turn any Software into an Agent Environmentcited by 24
  27. 2024DataEnvGym: Data Generation Agents in Teacher Environments with Student Feedbackcited by 23
  28. 2025FlairGPT: Repurposing LLMs for Interior Designscited by 22
  29. 2025HSM: Hierarchical Scene Motifs for Multi-Scale Indoor Scene Generationcited by 22
  30. 2024MegaSynth: Scaling Up 3D Scene Reconstruction with Synthesized Datacited by 21
  31. 2025LiteReality: Graphics-Ready 3D Scene Reconstruction from RGB-D Scanscited by 21
  32. 2024DeBaRA: Denoising-Based 3D Room Arrangement Generationcited by 20
  33. 2024Atlas3D: Physically Constrained Self-Supporting Text-to-3D for Simulation and Fabricationcited by 19
  34. 2024BlenderAlchemy: Editing 3D Graphics with Vision-Language Modelscited by 18
  35. 2026RHO: Your Coding Agent is Secretly a Roboticistcited by 18
  36. 2025A Survey of Robotic Navigation and Manipulation with Physics Simulators in the Era of Embodied AIcited by 16
  37. 2025Eval3D: Interpretable and Fine-grained Evaluation for 3D Generationcited by 15
  38. 2025Simulating the Visual World with Artificial Intelligence: A Roadmapcited by 15

50 most cited of 257 All 257 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026SceneFoundry: Generating Interactive Infinite 3D Worlds
  2. 2024Open-Universe Indoor Scene Generation using LLM Program Synthesis and Uncurated Object Databases
  3. 2026SceneSmith: Agentic Generation of Simulation-Ready Indoor Scenes
  4. 2026Hoi3DGen: Generating High-Quality Human-Object-Interactions in 3D
  5. 2026CoGen3D: An Agentic Human-AI Co-Design Pipeline for 3D Asset Generation for Virtual Reality
  6. 2026D3D-GEN: Robot-Aware Domain-Grounded Interactive 3D World Generation for Social Robotics ↗
  7. 2026RoboLayout: Differentiable 3D Scene Generation for Embodied Agents
  8. 2026Open-Vocabulary Functional 3D Human-Scene Interaction Generation