mlsanityData to 7 Oct 2026

LLMR: Real-time Prompting of Interactive Worlds using Large Language Models

arXiv 2309.12276 ↗First posted 21 Sep 2023cs.HC cs.AI cs.CLcited by 39

Not on the map. Why?

Abstract (arXiv)

We present Large Language Model for Mixed Reality (LLMR), a framework for the real-time creation and modification of interactive Mixed Reality experiences using LLMs. LLMR leverages novel strategies to tackle difficult cases where ideal training data is scarce, or where the design goal requires the synthesis of internal dynamics, intuitive analysis, or advanced interactivity. Our framework relies on text interaction and the Unity game engine. By incorporating techniques for scene understanding, task planning, self-debugging, and memory management, LLMR outperforms the standard GPT-4 by 4x in average error rate. We demonstrate LLMR's cross-platform interoperability with several example worlds, and evaluate it on a variety of creation and modification tasks to show that it can produce and edit diverse objects, tools, and scenes. Finally, we conducted a usability study (N=11) with a diverse set that revealed participants had positive experiences with the system and would use it again.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Builds on 38 of its 116 references that are in the map. First posted on arXiv 21 Sep 2023.

Builds on

38 of 116 references.

References · 38

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  3. 2021Program Synthesis with Large Language Modelscited by 3,509
  4. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  5. 2023A Survey of Large Language Modelscited by 2,537
  6. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  7. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  8. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,285
  9. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  10. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  11. 2023HexPlane: A Fast Representation for Dynamic Scenescited by 582
  12. 2023MemoryBank: Enhancing Large Language Models with Long-Term Memorycited by 539
26 more
  1. 2023Instruct-NeRF2NeRF: Editing 3D Scenes with Instructionscited by 482
  2. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 478
  3. 2023Shap-E: Generating Conditional 3D Implicit Functionscited by 379
  4. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
  5. 2023TEXTure: Text-Guided Texturing of 3D Shapescited by 303
  6. 2023SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planningcited by 289
  7. 2023Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Modelscited by 245
  8. 2023Text-To-4D Dynamic Scene Generationcited by 204
  9. 2022PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learningcited by 186
  10. 2023Augmenting Language Models with Long-Term Memorycited by 186
  11. 2023RoCo: Dialectic Multi-Robot Collaboration with Large Language Modelscited by 171
  12. 2023MotionGPT: Finetuned LLMs Are General-Purpose Motion Generatorscited by 145
  13. 2023Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenescited by 125
  14. 2023Large Language Models are Few-Shot Health Learnerscited by 115
  15. 2023HoloDiffusion: Training a 3D Diffusion Model using 2D Imagescited by 106
  16. 2021Jigsaw: Large Language Models meet Program Synthesiscited by 103
  17. 2023Embodied Task Planning with Large Language Modelscited by 84
  18. 2022Keep Me Updated! Memory Management in Long-term Conversationscited by 62
  19. 2023Recursively Summarizing Enables Long-Term Dialogue Memory in Large Language Modelscited by 62
  20. 20223DDesigner: Towards Photorealistic 3D Object Generation and Editing with Text-guided Diffusion Modelscited by 57
  21. 2020Do RNN and LSTM have Long Memory?cited by 47
  22. 2022PanGu-Coder: Program Synthesis with Function-Level Language Modelingcited by 47
  23. 2023Memory Sandbox: Transparent and Interactive Memory Management for Conversational Agentscited by 36
  24. 2023XAIR: A Framework of Explainable AI in Augmented Realitycited by 17
  25. 2022Steps towards prompt-based creation of virtual worldscited by 7
  26. 2021A Survey on Synchronous Augmented, Virtual and Mixed Reality Remote Collaboration Systemscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 14; 2025: 17; 2026: 7.

39 papers.

Sort:

By area:

  1. 2024Large Language Models and Games: A Survey and Roadmapcited by 91
  2. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  3. 2025BlenderGym: Benchmarking Foundational Model Systems for Graphics Editingcited by 27
  4. 2024BlenderAlchemy: Editing 3D Graphics with Vision-Language Modelscited by 18
  5. 2024Space3D-Bench: Spatial 3D Question Answering Benchmarkcited by 17
  6. 2025Understanding the LLM-ification of CHI: Unpacking the Impact of LLMs at CHI through a Systematic Literature Reviewcited by 16
  7. 2024DreamGarden: A Designer Assistant for Growing Games from a Single Promptcited by 10
  8. 2024GPT for Games: An Updated Scoping Review (2020-2024)cited by 10
  9. 2025MoVer: Motion Verification for Motion Graphics Animationscited by 10
  10. 2024How Secure Are Large Language Models (LLMs) for Navigation in Urban Environments?cited by 7
  11. 2025LLMER: Crafting Interactive Extended Reality Worlds with JSON Data Generated by Large Language Modelscited by 7
  12. 2025Explainable XR: Understanding User Behaviors of XR Environments using LLM-assisted Analytics Frameworkcited by 6
27 more
  1. 2024CUIfy the XR: An Open-Source Package to Embed LLM-powered Conversational Agents in XRcited by 5
  2. 2024A Study on Training and Developing Large Language Models for Behavior Tree Generationcited by 4
  3. 2024Analyzing Multimodal Interaction Strategies for LLM-Assisted Manipulation of 3D Scenescited by 4
  4. 2024SpaceBlender: Creating Context-Rich Collaborative Spaces Through Generative 3D Scene Blendingcited by 3
  5. 20253Dify: a Framework for Procedural 3D-CG Generation Assisted by LLMs Using MCP and RAGcited by 3
  6. 2025VULCAN: Tool-Augmented Multi Agents for Iterative 3D Object Arrangementcited by 3
  7. 2023InstructPipe: Generating Visual Blocks Pipelines with Human Instructions and LLMscited by 2
  8. 2024Everyday AR through AI-in-the-Loopcited by 2
  9. 2025Dreamcrafter: Immersive Editing of 3D Radiance Fields Through Flexible, Generative Inputs and Outputscited by 2
  10. 2024PanoTree: Autonomous Photo-Spot Explorer in Virtual Reality Scenescited by 1
  11. 2024Geometric Algebra Meets Large Language Models: Instruction-Based Transformations of Separate Meshes in 3D, Interactive and Controllable Scenescited by 1
  12. 2024SHAPE-IT: Exploring Text-to-Shape-Display for Generative Shape-Changing Behaviors with LLMscited by 1
  13. 2025AIvaluateXR: An Evaluation Framework for on-Device AI in XR with Benchmarking Resultscited by 1
  14. 2025Exploring Context-aware and LLM-driven Locomotion for Immersive Virtual Realitycited by 1
  15. 2025How LLMs are Shaping the Future of Virtual Realitycited by 1
  16. 2025PILAR: Personalizing Augmented Reality Interactions with LLM-based Human-Centric and Trustworthy Explanations for Daily Use Casescited by 1
  17. 2026When Generative AI Meets Extended Reality: Enabling Scalable and Natural Interactionscited by 1
  18. 2026Memento: Towards Proactive Visualization of Everyday Memories with Personal Wearable AR Assistantcited by 1
  19. 2026PRISM-XR: Empowering Privacy-Aware XR Collaboration with Multimodal Large Language Modelscited by 1
  20. 2026Self-Consistency for LLM-Based Motion Trajectory Generation and Verificationcited by 1
  21. 2025Text2VR: Automated instruction Generation in Virtual Reality using Large language Models for Assembly Taskcited by 0
  22. 2025Empowering Children to Create AI-Enabled Augmented Reality Experiencescited by 0
  23. 2025Phythesis: Physics-Guided Evolutionary Scene Synthesis for Energy-Efficient Data Center Design via LLMscited by 0
  24. 2025Exploring User Acceptance and Concerns toward LLM-powered Conversational Agents in Immersive Extended Realitycited by 0
  25. 2026ClickAIXR: On-Device Multimodal Vision-Language Interaction with Real-World Objects in Extended Realitycited by 0
  26. 2026From Prompt to Service: An SLM-Based Agent Orchestration Gateway for AI-Driven Virtual Worldscited by 0
  27. 2026AgentGarten: Code Worlds for Evolving Agentscited by 0