mlsanityData to 14 Oct 2026

Analyzing Multimodal Interaction Strategies for LLM-Assisted Manipulation of 3D Scenes

arXiv 2410.22177 ↗First posted 29 Oct 2024cs.HC cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

As more applications of large language models (LLMs) for 3D content for immersive environments emerge, it is crucial to study user behaviour to identify interaction patterns and potential barriers to guide the future design of immersive content creation and editing systems which involve LLMs. In an empirical user study with 12 participants, we combine quantitative usage data with post-experience questionnaire feedback to reveal common interaction patterns and key barriers in LLM-assisted 3D scene editing systems. We identify opportunities for improving natural language interfaces in 3D design tools and propose design recommendations for future LLM-integrated 3D content creation systems. Through an empirical study, we demonstrate that LLM-assisted interactive systems can be used productively in immersive environments.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 22 of its 52 references that are in the map. First posted on arXiv 29 Oct 2024.

Builds on

22 of 52 references.

References · 22

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  4. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  5. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  6. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 698
  7. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  8. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 615
  9. 2023AI Transparency in the Age of LLMs: A Human-Centered Research Roadmapcited by 123
  10. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Modelscited by 43
  11. 2023LLMR: Real-time Prompting of Interactive Worlds using Large Language Modelscited by 39
  12. 2024GazePointAR: A Context-Aware Multimodal Voice Assistant for Pronoun Disambiguation in Wearable Augmented Realitycited by 39
10 more
  1. 2024Embedding Large Language Models into Extended Reality: Opportunities and Challenges for Inclusion, Engagement, and Privacycited by 14
  2. 2024Exploring the Impact of AI Value Alignment in Collaborative Ideation: Effects on Perception, Ownership, and Outputcited by 12
  3. 2024Investigating Interaction Modes and User Agency in Human-LLM Collaboration for Domain-Specific Data Analysiscited by 12
  4. 2024VR-GPT: Visual Language Model for Intelligent Virtual Reality Applicationscited by 9
  5. 2022Steps towards prompt-based creation of virtual worldscited by 7
  6. 2022Co-Located Human-Human Interaction Analysis using Nonverbal Cues: A Surveycited by 6
  7. 2024How People Prompt to Create Interactive VR Scenescited by 6
  8. 2024MagicItem: Dynamic Behavior Design of Virtual Objects with Large Language Models in a Consumer Metaverse Platformcited by 4
  9. 2020Understanding Gesture and Speech Multimodal Interactions for Manipulation Tasks in Augmented Reality Using Unconstrained Elicitationcited by 3
  10. 2022Eliciting Multimodal Gesture+Speech Interactions in a Multi-Object Augmented Reality Environmentcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025SD-VLM: Spatial Measuring and Understanding with Depth-Encoded Vision-Language Modelscited by 25
  2. 2026Vinedresser3D: Agentic Text-guided 3D Editingcited by 3
  3. 2026From Pixels to Policies: Reinforcing Spatial Reasoning in Language Models for Content-Aware Layout Designcited by 1
  4. 2026SPHERE: Adaptive VR Indoor Scene Generation via LLM-Enhanced Spatial Preference Learning and Human-in-the-Loop RLcited by 0