mlsanityData to 7 Oct 2026

Steps towards prompt-based creation of virtual worlds

arXiv 2211.05875 ↗First posted 10 Nov 2022cs.HC cs.AI cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

Large language models trained for code generation can be applied to speaking virtual worlds into existence (creating virtual worlds). In this work we show that prompt-based methods can both accelerate in-VR level editing, as well as can become part of gameplay rather than just part of game development. As an example, we present Codex VR Pong which shows non-deterministic game mechanics using generative processes to not only create static content but also non-trivial interactions between 3D objects. This demonstration naturally leads to an integral discussion on how one would evaluate and benchmark experiences created by generative models - as there are no qualitative or quantitative metrics that apply in these scenarios. We conclude by discussing impending challenges of AI-assisted co-creation in VR.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 14 of its 46 references that are in the map. First posted on arXiv 10 Nov 2022.

Builds on

14 of 46 references.

References · 14

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  6. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  7. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  8. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  9. 2021FastNeRF: High-Fidelity Neural Rendering at 200FPScited by 510
  10. 2022GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Imagescited by 446
  11. 2021Animatable Neural Radiance Fields for Modeling Dynamic Human Bodiescited by 361
  12. 2021Neural Articulated Radiance Fieldcited by 196
2 more
  1. 2021Animatable Neural Radiance Fields from Monocular RGB Videoscited by 109
  2. 2021A Neural Network Solves, Explains, and Generates University Math Problems by Program Synthesis and Few-Shot Learning at Human Levelcited by 96

Built on by

Cited by year

Citing papers per year: 2023: 4; 2024: 3.

7 papers.

  1. 2023LLMR: Real-time Prompting of Interactive Worlds using Large Language Modelscited by 39
  2. 2024Large Language Models and Video Games: A Preliminary Scoping Reviewcited by 23
  3. 2023ArK: Augmented Reality with Knowledge Interactive Emergent Abilitycited by 5
  4. 2023Real-time Animation Generation and Control on Rigged Models via Large Language Modelscited by 5
  5. 2023Grimm in Wonderland: Prompt Engineering with Midjourney to Illustrate Fairytalescited by 4
  6. 2024Analyzing Multimodal Interaction Strategies for LLM-Assisted Manipulation of 3D Scenescited by 4
  7. 2024PanoTree: Autonomous Photo-Spot Explorer in Virtual Reality Scenescited by 1