# GazeVLM: A Vision-Language Model for Multi-Task Gaze Understanding

arXiv: https://arxiv.org/abs/2511.06348 · HTML: https://mlsanity.com/paper/2511.06348/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 25 of its 41 references that are in the map.
- First posted on arXiv 9 Nov 2025.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [Detecting Twenty-thousand Classes using Image-level Supervision](https://mlsanity.com/paper/2201.02605/) (2022)
- [Learning Rich Features from RGB-D Images for Object Detection and Segmentation](https://mlsanity.com/paper/1407.5736/) (2014)
- [Detecting Attended Visual Targets in Video](https://mlsanity.com/paper/2003.02501/) (2020)
- [Connecting Gaze, Scene, and Attention: Generalized Attention Estimation via Joint Modeling of Gaze and Scene Saliency](https://mlsanity.com/paper/1807.10437/) (2018)
- [End-to-End Human-Gaze-Target Detection with Transformers](https://mlsanity.com/paper/2203.10433/) (2022)
- [Gaze-LLE: Gaze Target Estimation via Large-Scale Learned Encoders](https://mlsanity.com/paper/2412.09586/) (2024)
- [ChildPlay: A New Benchmark for Understanding Children's Gaze Behaviour](https://mlsanity.com/paper/2307.01630/) (2023)
- [Automatic Gaze Analysis: A Survey of Deep Learning based Approaches](https://mlsanity.com/paper/2108.05479/) (2021)
- [A Modular Multimodal Architecture for Gaze Target Prediction: Application to Privacy-Sensitive Settings](https://mlsanity.com/paper/2307.05158/) (2023)
- [Voila-A: Aligning Vision-Language Models with User's Gaze Attention](https://mlsanity.com/paper/2401.09454/) (2023)
- [Multimodal Across Domains Gaze Target Detection](https://mlsanity.com/paper/2208.10822/) (2022)
- [Patch-level Gaze Distribution Prediction for Gaze Following](https://mlsanity.com/paper/2211.11062/) (2022)
- [Multimodal Integration of Human-Like Attention in Visual Question Answering](https://mlsanity.com/paper/2109.13139/) (2021)
- [GaTector: A Unified Framework for Gaze Object Prediction](https://mlsanity.com/paper/2112.03549/) (2021)
- [GazeOnce: Real-Time Multi-Person Gaze Estimation](https://mlsanity.com/paper/2204.09480/) (2022)
- [ViTGaze: Gaze Following with Interaction Features in Vision Transformers](https://mlsanity.com/paper/2403.12778/) (2024)
- [Exploring the Zero-Shot Capabilities of Vision-Language Models for Improving Gaze Following](https://mlsanity.com/paper/2406.03907/) (2024)
- [Spatio-Temporal Attention and Gaussian Processes for Personalized Video Gaze Estimation](https://mlsanity.com/paper/2404.05215/) (2024)
- [Boosting Gaze Object Prediction via Pixel-level Supervision from Vision Foundation Model](https://mlsanity.com/paper/2408.01044/) (2024)
- [Merging Multiple Datasets for Improved Appearance-Based Gaze Estimation](https://mlsanity.com/paper/2409.00912/) (2024)
- [Leveraging Multi-Modal Saliency and Fusion for Gaze Target Detection](https://mlsanity.com/paper/2504.19271/) (2025)

## Built on by (most cited first, 9 of 9)

- [Gaze-Regularized Vision-Language-Action Models for Robotic Manipulation](https://mlsanity.com/paper/2603.23202/) (2026)
- [GRASP: Learning to Ground Social Reasoning in Multi-Person Non-Verbal Interactions](https://mlsanity.com/paper/2605.15764/) (2026)
- [Learning to See What You Need: Gaze Attention for Multimodal Large Language Models](https://mlsanity.com/paper/2605.13080/) (2026)
- [Eyes on VLM: Benchmarking Gaze Following and Social Gaze Prediction in Vision Language Models](https://mlsanity.com/paper/2605.19859/) (2026)
- [Leveraging Vision-Language Models to Detect Attention in Educational Videos](https://mlsanity.com/paper/2605.20211/) (2026)
- [OmniGF: A Dual-Branch Vision-Language Framework for Unified Gaze Following](https://mlsanity.com/paper/2605.26399/) (2026)
- [Multi-scale Object-Aware Gaze Estimation via Geometric Reasoning](https://mlsanity.com/paper/2606.29334/) (2026)
- [Open-Vocabulary Gaze Object Prediction: Benchmark and Method](https://mlsanity.com/paper/2607.18827/) (2026)
- [TransGaze-Object: Transformer Based Driver Gaze Object Prediction Framework in Real Driving](https://mlsanity.com/paper/2609.10139/) (2026)

## Abstract (arXiv)

Gaze understanding unifies the detection of people, their gaze targets, and objects of interest into a single framework, offering critical insight into visual attention and intent estimation. Although prior research has modelled gaze cues in visual scenes, a unified system is still needed for gaze understanding using both visual and language prompts. This paper introduces GazeVLM, a novel Vision-Language Model (VLM) for multi-task gaze understanding in images, addressing person detection, gaze target detection, and gaze object identification. While other transformer-based methods exist for gaze analysis, GazeVLM represents, to our knowledge, the first application of a VLM to these combined tasks, allowing for selective execution of each task. Through the integration of visual (RGB and depth) and textual modalities, our ablation study on visual input combinations revealed that a fusion of RGB images with HHA-encoded depth maps, guided by text prompts, yields superior performance. We also introduce an object-level gaze detection metric for gaze object identification (APob). Through experiments, GazeVLM demonstrates significant improvements, notably achieving state-of-the-art evaluation scores on GazeFollow and VideoAttentionTarget datasets.