# Eye Gaze Tells You Where to Compute: Gaze-Driven Efficient VLMs

arXiv: https://arxiv.org/abs/2509.16476 · HTML: https://mlsanity.com/paper/2509.16476/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 17 of its 27 references that are in the map.
- First posted on arXiv 20 Sep 2025.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 17 of 17)

- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference](https://mlsanity.com/paper/2410.04417/) (2024)
- [MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases](https://mlsanity.com/paper/2402.14905/) (2024)
- [LongVLM: Efficient Long Video Understanding via Large Language Models](https://mlsanity.com/paper/2404.03384/) (2024)
- [Vision-Language Models for Medical Report Generation and Visual Question Answering: A Review](https://mlsanity.com/paper/2403.02469/) (2024)
- [MMTok: Multimodal Coverage Maximization for Efficient Inference of VLMs](https://mlsanity.com/paper/2508.18264/) (2025)
- [3ET: Efficient Event-based Eye Tracking using a Change-Based ConvLSTM Network](https://mlsanity.com/paper/2308.11771/) (2023)
- [Voila-A: Aligning Vision-Language Models with User's Gaze Attention](https://mlsanity.com/paper/2401.09454/) (2023)
- [G-VOILA: Gaze-Facilitated Information Querying in Daily Scenarios](https://mlsanity.com/paper/2405.07652/) (2024)
- [FACET: Fast and Accurate Event-Based Eye Tracking Using Ellipse Modeling for Extended Reality](https://mlsanity.com/paper/2409.15584/) (2024)
- [OmniVLM: A Token-Compressed, Sub-Billion-Parameter Vision-Language Model for Efficient On-Device Inference](https://mlsanity.com/paper/2412.11475/) (2024)
- [GazeHTA: End-to-end Gaze Target Detection with Head-Target Association](https://mlsanity.com/paper/2404.10718/) (2024)
- [Event-Priori-Based Vision-Language Model for Efficient Visual Understanding](https://mlsanity.com/paper/2506.07627/) (2025)
- [AdaFV: Rethinking of Visual-Language alignment for VLM acceleration](https://mlsanity.com/paper/2501.09532/) (2025)
- [Integrating Cognitive Processing Signals into Language Models: A Review of Advances, Applications and Future Directions](https://mlsanity.com/paper/2504.06843/) (2025)

## Built on by (most cited first, 3 of 3)

- [How Much Does It Cost to Answer My Question? Benchmarking Cloud VLM-based VQA Systems](https://mlsanity.com/paper/2608.07861/) (2026)
- [EgoGazeLite: On-Device Egocentric Gaze Prediction for Token-Efficient Multimodal LLM Video Input](https://mlsanity.com/paper/2608.15614/) (2026)
- [OpenVAM: Open-World Visual Attention Modeling with VLMs](https://mlsanity.com/paper/2609.31364/) (2026)

## Abstract (arXiv)

Vision-Language Models (VLMs) deliver impressive performance in understanding visual content with language instructions. However, redundancy in vision tokens results in the degenerated inference efficiency of VLMs, which hinders real-time use on edge consumer devices such as AR/VR devices. Existing efficiency methods commonly prune visual tokens using learned saliency, sparse attention schedules, or controller policies, but they often require architectural modification or access to intermediate activations. These pipelines add inference-time modules that increase compute and memory and often lead to an accuracy trade-off. Moreover, they also suffer from misalignment between the prompts and the region of interest in the images. Without human guidance, the model may focus on the wrong regions and miss small, high-frequency details when prompts or scenes change. In this paper, we propose GazeVLM, a training-free framework that uses the human eye gaze as a natural supervisory signal to allocate computation where it matters. By extracting gaze-driven regions of interest (ROIs) and optionally combining them with a low-resolution global view, GazeVLM mimics fovea-periphery perception to cut redundant visual tokens while preserving task-relevant details. We evaluate the visual question answering tasks on Qwen2.5-VL-3B/7B on the VOILA-COCO benchmark with human gaze. Quality of the answer is assessed by GPT-4o pairwise judging and a weighted score over coverage, accuracy, details, and fluency. Efficiency is measured by token counts and FLOPs. GazeVLM reduces visual tokens by up to 93.1%, total tokens by up to 59.6%, and FLOPs by 50%, while keeping better answer quality relative to full-resolution baselines. Our results show that aligning model computation with human gaze offers a simple, plug-and-play path toward efficient VLM inference on consumer devices.