SIAgent: Spatial Interaction Agent via LLM-powered Eye-Hand Motion Intent Understanding in VR
On the map: LLM agents and tool use → Multimodal and embodied agents
Abstract (arXiv)
Eye-hand coordinated interaction is becoming a mainstream interaction modality in Virtual Reality (VR) user interfaces.Current paradigms for this multimodal interaction require users to learn predefined gestures and memorize multiple gesture-task associations, which can be summarized as an ``Operation-to-Intent" paradigm. This paradigm increases users' learning costs and has low interaction error tolerance. In this paper, we propose SIAgent, a novel "Intent-to-Operation" framework allowing users to express interaction intents through natural eye-hand motions based on common sense and habits. Our system features two main components: (1) intent recognition that translates spatial interaction data into natural language and infers user intent, and (2) agent-based execution that generates an agent to execute corresponding tasks. This eliminates the need for gesture memorization and accommodates individual motion preferences with high error tolerance. We conduct two user studies across over 60 interaction tasks, comparing our method with two "Operation-to-Intent" techniques. Results show our method achieves higher intent recognition accuracy than gaze + pinch interaction (97.2% vs 93.1%) while reducing arm fatigue and improving usability, and user preference. Another study verifies the function of eye gaze and hand motion channels in intent recognition. Our work offers valuable insights into enhancing VR interaction intelligence through intent-driven design. Our source code and LLM prompts will be made available upon publication.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 12 of its 57 references that are in the map. First posted on arXiv 28 Feb 2026.
Builds on
12 of 57 references.
References · 12
- 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
- 2023When Large Language Models Meet Personalization: Perspectives of Challenges and Opportunitiescited by 90
- 2024Human-Object Interaction from Human-Level Instructionscited by 64
- 2024Autonomous Character-Scene Interaction Synthesis from Text Instructioncited by 56
- 2024GazePointAR: A Context-Aware Multimodal Voice Assistant for Pronoun Disambiguation in Wearable Augmented Realitycited by 39
- 2024SocialMind: LLM-based Proactive AR Social Assistive System with Human-like Perception for In-situ Live Interactionscited by 25
- 2024JointDreamer: Ensuring Geometry Consistency and Text Congruence in Text-to-3D Generation via Joint Score Distillationcited by 11
- 2025GazeLLM: Multimodal LLMs incorporating Human Visual Attentioncited by 9
- 2023DexCatch: Learning to Catch Arbitrary Objects with Dexterous Handscited by 8
- 2024Augmented Conversation with Embedded Speech-Driven On-the-Fly Referencing in ARcited by 4
- 2024Robust Dual-Modal Speech Keyword Spotting for XR Headsetscited by 2
- 2022Gaze-Vergence-Controlled See-Through Vision in Augmented Realitycited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026SARAH: Spatially Aware Real-time Agentic Humans
- 2023GestureGPT: Toward Zero-Shot Free-Form Hand Gesture Understanding with Large Language Model Agents
- 2025IntentionVLA: Generalizable and Efficient Embodied Intention Reasoning for Human-Robot Interaction
- 2025Plug-and-Play Clarifier: A Zero-Shot Multimodal Framework for Egocentric Intent Disambiguation
- 2026IntentionNav: A Benchmark for Intent-Driven Object Navigation from Implicit Human Instruction
- 2026LightNav-0: Eliciting VLM Spatial Intelligence for Generalist Embodied Navigation
- 2026Pointing-VLA: Typed Spatial Grounding Interfaces for Vision-Language-Action Manipulation
- 2025ContextAgent: Context-Aware Proactive LLM Agents with Open-World Sensory Perceptions