mlsanityData to 7 Oct 2026

DreamGarden: A Designer Assistant for Growing Games from a Single Prompt

arXiv 2410.01791 ↗First posted 2 Oct 2024cs.HC cs.AI cs.CLcited by 10

Not on the map. Why?

Abstract (arXiv)

Coding assistants are increasingly leveraged in game design, both generating code and making high-level plans. To what degree can these tools align with developer workflows, and what new modes of human-computer interaction can emerge from their use? We present DreamGarden, an AI system capable of assisting with the development of diverse game environments in Unreal Engine. At the core of our method is an LLM-driven planner, capable of breaking down a single, high-level prompt -- a dream, memory, or imagined scenario provided by a human user -- into a hierarchical action plan, which is then distributed across specialized submodules facilitating concrete implementation. This system is presented to the user as a garden of plans and actions, both growing independently and responding to user intervention via seed prompts, pruning, and feedback. Through a user study, we explore design implications of this system, charting courses for future work in semi-autonomous assistants and open-ended simulation design.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 44 of its 134 references that are in the map. First posted on arXiv 2 Oct 2024.

Builds on

44 of 134 references.

References · 44

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  4. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  5. 2019A Comprehensive Survey on Graph Neural Networkscited by 2,906
  6. 2018World Modelscited by 1,444
  7. 2020Mastering Atari with Discrete World Modelscited by 990
  8. 2023Objaverse-XL: A Universe of 10M+ 3D Objectscited by 767
  9. 2024Genie: Generative Interactive Environmentscited by 706
  10. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 484
  11. 2023TEXTure: Text-Guided Texturing of 3D Shapescited by 303
  12. 2024Diffusion for World Modeling: Visual Details Matter in Ataricited by 285
32 more
  1. 2024Diffusion Models Are Real-Time Game Enginescited by 282
  2. 2020Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Designcited by 260
  3. 2023Text2Tex: Text-driven Texture Synthesis via Diffusion Modelscited by 234
  4. 2024CodeGemma: Open Code Models Based on Gemmacited by 145
  5. 2017Procedural Content Generation via Machine Learning (PCGML)cited by 136
  6. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 136
  7. 2024Large Language Models and Games: A Survey and Roadmapcited by 91
  8. 2020PCGRL: Procedural Content Generation via Reinforcement Learningcited by 90
  9. 2023Level Generation Through Large Language Modelscited by 68
  10. 2024Scaling Instructable Agents Across Many Simulated Worldscited by 67
  11. 2024OMNI-EPIC: Open-endedness via Models of human Notions of Interestingness with Environments Programmed in Codecited by 67
  12. 2023MarioGPT: Open-Ended Text2Level Generation through Large Language Modelscited by 63
  13. 2018Lenia - Biology of Artificial Lifecited by 41
  14. 20223DALL-E: Integrating Text-to-Image AI in 3D Design Workflowscited by 41
  15. 2021Learning Controllable Content Generatorscited by 40
  16. 2023LLMR: Real-time Prompting of Interactive Worlds using Large Language Modelscited by 39
  17. 2020TOAD-GAN: Coherent Style Level Generation from a Single Examplecited by 22
  18. 2023ChatGPT and Other Large Language Models as Evolutionary Engines for Online Interactive Collaborative Game Designcited by 20
  19. 2022Learning Controllable 3D Level Generatorscited by 19
  20. 2020Video Game Level Repair via Mixed Integer Linear Programmingcited by 18
  21. 2024Word2World: Generating Stories and Worlds through Large Language Modelscited by 18
  22. 2020Learning to Generate Levels From Nothingcited by 17
  23. 2024DreamCraft: Text-Guided Generation of Functional 3D Environments in Minecraftcited by 17
  24. 2018Automated Game Design via Conceptual Expansioncited by 15
  25. 2023The Five-Dollar Model: Generating Game Maps and Sprites from Sentence Embeddingscited by 14
  26. 2020Towards Game Design via Creative Machine Learning (GDCML)cited by 12
  27. 2024Toward Artificial Open-Ended Evolution within Lenia using Quality-Diversitycited by 12
  28. 2023Style2Fab: Functionality-Aware Segmentation for Fabricating Personalized 3D Models with Generative AIcited by 11
  29. 2024GPT for Games: A Scoping Review (2020-2023)cited by 10
  30. 2023Real-time Animation Generation and Control on Rigged Models via Large Language Modelscited by 5
  31. 2017Automated Game Design Learningcited by 3
  32. 2022Aesthetic Bot: Interactively Evolving Game Maps on Twittercited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 5.

10 papers.

  1. 2026AutoUE: Automated Generation of 3D Games in Unreal Engine via Multi-Agent Systemscited by 7
  2. 2026JAMER: Project-Level Code Framework Dataset and Benchmark on Professional Game Enginescited by 5
  3. 2025PuzzleJAX: A Benchmark for Reasoning and Learningcited by 4
  4. 2025Video Game Level Design as a Multi-Agent Reinforcement Learning Problemcited by 1
  5. 2025Diverse Planning with Simulators via Linear Temporal Logiccited by 1
  6. 2025AnimAgents: Coordinating Multi-Stage Animation Pre-Production with Human-Multi-Agent Collaborationcited by 1
  7. 2025Elsewise: Authoring Open-ended Interactive Narrative with Possibility Space Visualizationcited by 0
  8. 2026High Dimensional Procedural Content Generationcited by 0
  9. 2026SWE-Game: Can Coding Agents Build the Games We Want?cited by 0
  10. 2026NarrativeSteward: Coordinating Delegation, Guidance, and Verification in Agent-Assisted Interactive Narrative Authoringcited by 0