# GazePointAR: A Context-Aware Multimodal Voice Assistant for Pronoun Disambiguation in Wearable Augmented Reality

arXiv: https://arxiv.org/abs/2404.08213 · HTML: https://mlsanity.com/paper/2404.08213/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Apr 2024.

## Built on by (most cited first, 30 of 39)

- [LION-FS: Fast & Slow Video-Language Thinker as Online Video Assistant](https://mlsanity.com/paper/2503.03663/) (2025)
- [Securing Large Language Models: Addressing Bias, Misinformation, and Prompt Attacks](https://mlsanity.com/paper/2409.08087/) (2024)
- [Device-Cloud Collaborative LLM Inference with Multi-Modal, Multi-Task, Multi-Turn Conversations](https://mlsanity.com/paper/2502.11007/) (2025)
- [Augmented Object Intelligence with XR-Objects](https://mlsanity.com/paper/2404.13274/) (2024)
- [Vid2Coach: Transforming How-To Videos into Task Assistants](https://mlsanity.com/paper/2506.00717/) (2025)
- [OmniQuery: Contextually Augmenting Captured Multimodal Memory to Enable Personal Question Answering](https://mlsanity.com/paper/2409.08250/) (2024)
- [VisionClaw: Always-On AI Agents through Smart Glasses](https://mlsanity.com/paper/2604.03486/) (2026)
- [Evaluating Scene-based In-Situ Item Labeling for Immersive Conversational Recommendation](https://mlsanity.com/paper/2604.09698/) (2026)
- [Analyzing Multimodal Interaction Strategies for LLM-Assisted Manipulation of 3D Scenes](https://mlsanity.com/paper/2410.22177/) (2024)
- [Gesturing Toward Abstraction: Multimodal Convention Formation in Collaborative Physical Tasks](https://mlsanity.com/paper/2602.08914/) (2026)
- [Cross-Format Retrieval-Augmented Generation in XR with LLMs for Context-Aware Maintenance Assistance](https://mlsanity.com/paper/2502.15604/) (2025)
- [SpeechLess: Micro-utterance with Personalized Spatial Memory-aware Assistant in Everyday Augmented Reality](https://mlsanity.com/paper/2602.00793/) (2026)
- [Seeing Eye to Eye: Enabling Cognitive Alignment Through Shared First-Person Perspective in Human-AI Collaboration](https://mlsanity.com/paper/2603.12701/) (2026)
- [Everyday AR through AI-in-the-Loop](https://mlsanity.com/paper/2412.12681/) (2024)
- [Beyond Language: Grounding Referring Expressions with Hand Pointing in Egocentric Vision](https://mlsanity.com/paper/2603.26646/) (2026)
- [Grounding Task Assistance with Multimodal Cues from a Single Demonstration](https://mlsanity.com/paper/2505.01578/) (2025)
- [Reality Proxy: Fluid Interactions with Real-World Objects in MR via Abstract Representations](https://mlsanity.com/paper/2507.17248/) (2025)
- [StepWrite: Adaptive Planning for Speech-Driven Text Generation](https://mlsanity.com/paper/2508.04011/) (2025)
- [PILAR: Personalizing Augmented Reality Interactions with LLM-based Human-Centric and Trustworthy Explanations for Daily Use Cases](https://mlsanity.com/paper/2512.17172/) (2025)
- [Memento: Towards Proactive Visualization of Everyday Memories with Personal Wearable AR Assistant](https://mlsanity.com/paper/2601.17622/) (2026)
- [From Speech-to-Spatial: Grounding Utterances on A Live Shared View with Augmented Reality](https://mlsanity.com/paper/2602.03059/) (2026)
- [ARGaze: Autoregressive Transformers for Online Egocentric Gaze Estimation](https://mlsanity.com/paper/2602.05132/) (2026)
- [Alfa: Attentive Low-Rank Filter Adaptation for Structure-Aware Cross-Domain Personalized Gaze Estimation](https://mlsanity.com/paper/2603.08445/) (2026)
- [Do You See What I Am Pointing At? Gesture-Based Egocentric Video Question Answering](https://mlsanity.com/paper/2603.12533/) (2026)
- [R4DSG: Relative 4D Scene Graph Memory for Object-Centric Question Answering in Long Egocentric Video](https://mlsanity.com/paper/2608.11017/) (2026)
- [RealitySummary: Exploring On-Demand Mixed Reality Text Summarization and Question Answering using Large Language Models](https://mlsanity.com/paper/2405.18620/) (2024)
- ["Does the cafe entrance look accessible? Where is the door?" Towards Geospatial AI Agents for Visual Inquiries](https://mlsanity.com/paper/2508.15752/) (2025)
- [Resolving Ambiguity in Gaze-Facilitated Visual Assistant Interaction Paradigm](https://mlsanity.com/paper/2509.21980/) (2025)
- [SIAgent: Spatial Interaction Agent via LLM-powered Eye-Hand Motion Intent Understanding in VR](https://mlsanity.com/paper/2603.00522/) (2026)
- [ClickAIXR: On-Device Multimodal Vision-Language Interaction with Real-World Objects in Extended Reality](https://mlsanity.com/paper/2604.04905/) (2026)

## Abstract (arXiv)

Voice assistants (VAs) like Siri and Alexa are transforming human-computer interaction; however, they lack awareness of users' spatiotemporal context, resulting in limited performance and unnatural dialogue. We introduce GazePointAR, a fully-functional context-aware VA for wearable augmented reality that leverages eye gaze, pointing gestures, and conversation history to disambiguate speech queries. With GazePointAR, users can ask "what's over there?" or "how do I solve this math problem?" simply by looking and/or pointing. We evaluated GazePointAR in a three-part lab study (N=12): (1) comparing GazePointAR to two commercial systems; (2) examining GazePointAR's pronoun disambiguation across three tasks; (3) and an open-ended phase where participants could suggest and try their own context-sensitive queries. Participants appreciated the naturalness and human-like nature of pronoun-driven queries, although sometimes pronoun use was counter-intuitive. We then iterated on GazePointAR and conducted a first-person diary study examining how GazePointAR performs in-the-wild. We conclude by enumerating limitations and design considerations for future context-aware VAs.