# Object Referring-Guided Scanpath Prediction with Perception-Enhanced Vision-Language Models

arXiv: https://arxiv.org/abs/2604.20361 · HTML: https://mlsanity.com/paper/2604.20361/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 15 of its 39 references that are in the map.
- First posted on arXiv 22 Apr 2026.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling](https://mlsanity.com/paper/2412.05271/) (2024)
- [ERNIE: Enhanced Representation through Knowledge Integration](https://mlsanity.com/paper/1904.09223/) (2019)
- [ScanGAN360: A Generative Model of Realistic Scanpaths for 360circ Images](https://mlsanity.com/paper/2103.13922/) (2021)
- [Connecting What to Say With Where to Look by Modeling Human Attention Traces](https://mlsanity.com/paper/2105.05964/) (2021)
- [Understanding Teacher Gaze Patterns for Robot Learning](https://mlsanity.com/paper/1907.07202/) (2019)
- [Trends, Applications, and Challenges in Human Attention Modelling](https://mlsanity.com/paper/2402.18673/) (2024)
- [HOI-Ref: Hand-Object Interaction Referral in Egocentric Vision](https://mlsanity.com/paper/2404.09933/) (2024)
- [StreamGaze: Gaze-Guided Temporal Reasoning and Proactive Understanding in Streaming Videos](https://mlsanity.com/paper/2512.01707/) (2025)
- [Object Referring in Visual Scene with Spoken Language](https://mlsanity.com/paper/1711.03800/) (2017)

## Abstract (arXiv)

Object Referring-guided Scanpath Prediction (ORSP) aims to predict the human attention scanpath when they search for a specific target object in a visual scene according to a linguistic description describing the object. Multimodal information fusion is a key point of ORSP. Therefore, we propose a novel model, ScanVLA, to first exploit a Vision-Language Model (VLM) to extract and fuse inherently aligned visual and linguistic feature representations from the input image and referring expression. Next, to enhance the ScanVLA's perception of fine-grained positional information, we not only propose a novel History Enhanced Scanpath Decoder (HESD) that directly takes historical fixations' position information as input to help predict a more reasonable position for the current fixation, but also adopt a frozen Segmentation LoRA as an auxiliary component to help localize the referred object more precisely, which improves the scanpath prediction task without incurring additional large computational and time costs. Extensive experimental results demonstrate that ScanVLA can significantly outperform existing scanpath prediction methods under object referring.