# GazeGPT: Augmenting Human Capabilities using Gaze-contingent Contextual AI for Smart Eyewear

arXiv: https://arxiv.org/abs/2401.17217 · HTML: https://mlsanity.com/paper/2401.17217/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 10 of its 34 references that are in the map.
- First posted on arXiv 30 Jan 2024.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 10 of 10)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [Ferret: Refer and Ground Anything Anywhere at Any Granularity](https://mlsanity.com/paper/2310.07704/) (2023)
- [Multimodal Foundation Models: From Specialists to General-Purpose Assistants](https://mlsanity.com/paper/2309.10020/) (2023)
- [Project Aria: A New Tool for Egocentric Multi-Modal AI Research](https://mlsanity.com/paper/2308.13561/) (2023)
- [AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Model](https://mlsanity.com/paper/2309.16058/) (2023)
- [Gaze-Contingent Ocular Parallax Rendering for Virtual Reality](https://mlsanity.com/paper/1906.09740/) (2019)
- [Noise-based Enhancement for Foveated Rendering](https://mlsanity.com/paper/2204.04455/) (2022)
- [Perceptual Rasterization for Head-mounted Display Image Synthesis](https://mlsanity.com/paper/1806.05385/) (2018)

## Built on by (most cited first, 16 of 16)

- [In the Eye of MLLM: Benchmarking Egocentric Video Intent Understanding with Gaze-Guided Prompting](https://mlsanity.com/paper/2509.07447/) (2025)
- [Gazing Into Missteps: Leveraging Eye-Gaze for Unsupervised Mistake Detection in Egocentric Videos of Skilled Human Activities](https://mlsanity.com/paper/2406.08379/) (2024)
- [GazeLLM: Multimodal LLMs incorporating Human Visual Attention](https://mlsanity.com/paper/2504.00221/) (2025)
- [Reading Recognition in the Wild](https://mlsanity.com/paper/2505.24848/) (2025)
- [VisionClaw: Always-On AI Agents through Smart Glasses](https://mlsanity.com/paper/2604.03486/) (2026)
- [Segment This Thing: Foveated Tokenization for Efficient Point-Prompted Segmentation](https://mlsanity.com/paper/2506.11131/) (2025)
- [SkillSight: Efficient First-Person Skill Assessment with Gaze](https://mlsanity.com/paper/2511.19629/) (2025)
- [EgoProx: Evaluating MLLMs on Egocentric 3D Proximity Reasoning Across a Cognitive Hierarchy](https://mlsanity.com/paper/2605.24456/) (2026)
- [Eye Gaze as a Signal for Conveying User Attention in Contextual AI Systems](https://mlsanity.com/paper/2501.13878/) (2025)
- [Reflections and New Directions for Human-Centered Large Language Models](https://mlsanity.com/paper/2605.06901/) (2026)
- [Grounding Task Assistance with Multimodal Cues from a Single Demonstration](https://mlsanity.com/paper/2505.01578/) (2025)
- [SteROI-D: System Design and Mapping for Stereo Depth Inference on Regions of Interest](https://mlsanity.com/paper/2502.09528/) (2025)
- [Resolving Ambiguity in Gaze-Facilitated Visual Assistant Interaction Paradigm](https://mlsanity.com/paper/2509.21980/) (2025)
- [EyeCue: Driver Cognitive Distraction Detection via Gaze-Empowered Egocentric Video Understanding](https://mlsanity.com/paper/2605.07859/) (2026)
- [AmbientEye: A Dataset for Pupil Segmentation under Natural Ambient Infrared Illumination](https://mlsanity.com/paper/2606.03774/) (2026)
- [EgoGazeLite: On-Device Egocentric Gaze Prediction for Token-Efficient Multimodal LLM Video Input](https://mlsanity.com/paper/2608.15614/) (2026)

## Abstract (arXiv)

Multimodal large language models (LMMs) excel in world knowledge and problem-solving abilities. Through the use of a world-facing camera and contextual AI, emerging smart accessories aim to provide a seamless interface between humans and LMMs. Yet, these wearable computing systems lack an understanding of the user's attention. We introduce GazeGPT as a new user interaction paradigm for contextual AI. GazeGPT uses eye tracking to help the LMM understand which object in the world-facing camera view a user is paying attention to. Using extensive user evaluations, we show that this gaze-contingent mechanism is a faster and more accurate pointing mechanism than alternatives; that it augments human capabilities by significantly improving their accuracy in a dog-breed classification task; and that it is consistently ranked as more natural than head- or body-driven selection mechanisms for contextual AI. Moreover, we prototype a variety of application scenarios that suggest GazeGPT could be of significant value to users as part of future AI-driven personal assistants.