# From Scene to Object: Text-Guided Dual-Gaze Prediction

arXiv: https://arxiv.org/abs/2604.20191 · HTML: https://mlsanity.com/paper/2604.20191/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 22 of its 34 references that are in the map.
- First posted on arXiv 22 Apr 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Driving VLMs and open-vocabulary detection](https://mlsanity.com/mllm/driving-vlms-and-open-vocabulary-detection/)

## Builds on (most cited first, 22 of 22)

- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Models](https://mlsanity.com/paper/2505.19223/) (2025)
- [Gaze360: Physically Unconstrained Gaze Estimation in the Wild](https://mlsanity.com/paper/1910.10088/) (2019)
- [SimLingo: Vision-Only Closed-Loop Autonomous Driving with Language-Action Alignment](https://mlsanity.com/paper/2503.09594/) (2025)
- [Predicting the Driver's Focus of Attention: the DR(eye)VE Project](https://mlsanity.com/paper/1705.03854/) (2017)
- [Predicting Driver Attention in Critical Situations](https://mlsanity.com/paper/1711.06406/) (2017)
- [Deep Gaze I: Boosting Saliency Prediction with Feature Maps Trained on ImageNet](https://mlsanity.com/paper/1411.1045/) (2014)
- [DADA-2000: Can Driving Accident be Predicted by Driver Attention? Analyzed by A Benchmark](https://mlsanity.com/paper/1904.12634/) (2019)
- [DeepGaze IIE: Calibrated prediction in and out-of-domain for state-of-the-art saliency modeling](https://mlsanity.com/paper/2105.12441/) (2021)
- [A Deep Multi-Level Network for Saliency Prediction](https://mlsanity.com/paper/1609.01064/) (2016)
- [MEDIRL: Predicting the Visual Attention of Drivers via Maximum Entropy Deep Inverse Reinforcement Learning](https://mlsanity.com/paper/1912.07773/) (2019)
- [Unsupervised Self-Driving Attention Prediction via Uncertainty Mining and Knowledge Embedding](https://mlsanity.com/paper/2303.09706/) (2023)
- [GazeXplain: Learning to Predict Natural Language Explanations of Visual Scanpaths](https://mlsanity.com/paper/2408.02788/) (2024)
- [FBLNet: FeedBack Loop Network for Driver Attention Prediction](https://mlsanity.com/paper/2212.02096/) (2022)
- [Where, What, Why: Towards Explainable Driver Attention Prediction](https://mlsanity.com/paper/2506.23088/) (2025)
- [MAAD: A Model and Dataset for "Attended Awareness" in Driving](https://mlsanity.com/paper/2110.08610/) (2021)
- [Visual Saliency Prediction Using a Mixture of Deep Neural Networks](https://mlsanity.com/paper/1702.00372/) (2017)
- [SCOUT+: Towards Practical Task-Driven Drivers' Gaze Prediction](https://mlsanity.com/paper/2404.08756/) (2024)
- [SalM2: An Extremely Lightweight Saliency Mamba Model for Real-Time Cognitive Awareness of Driver Attention](https://mlsanity.com/paper/2502.16214/) (2025)
- [Spatial-aware Vision Language Model for Autonomous Driving](https://mlsanity.com/paper/2512.24331/) (2025)
- [FSDAM: Few-Shot Driving Attention Modeling via Vision-Language Coupling](https://mlsanity.com/paper/2511.12708/) (2025)

## Built on by (most cited first, 1 of 1)

- [FocusDrive: Reasoning with Visual Focus for Autonomous Driving](https://mlsanity.com/paper/2609.33190/) (2026)

## Abstract (arXiv)

Interpretable driver attention prediction is crucial for human-like autonomous driving. However, existing datasets provide only scene-level global gaze rather than fine-grained object-level annotations, inherently failing to support text-grounded cognitive modeling. Consequently, while Vision-Language Models (VLMs) hold great potential for semantic reasoning, this critical data limitations leads to severe text-vision decoupling and visual-bias hallucinations. To break this bottleneck and achieve precise object-level attention prediction, this paper proposes a novel dual-branch gaze prediction framework, establishing a complete paradigm from data construction to model architecture. First, we construct G-W3DA, a object-level driver attention dataset. By integrating a multimodal large language model with the Segment Anything Model 3 (SAM3), we decouple macroscopic heatmaps into object-level masks under rigorous cross-validation, fundamentally eliminating annotation hallucinations. Building upon this high-quality data foundation, we propose the DualGaze-VLM architecture. This architecture extracts the hidden states of semantic queries and dynamically modulates visual features via a Condition-Aware SE-Gate, achieving intent-driven precise spatial anchoring. Extensive experiments on the W3DA benchmark demonstrate that DualGaze-VLM consistently surpasses existing state-of-the-art (SOTA) models in spatial alignment metrics, notably achieving up to a 17.8% improvement in Similarity (SIM) under safety-critical scenarios. Furthermore, a visual Turing test reveals that the attention heatmaps generated by DualGaze-VLM are perceived as authentic by 88.22% of human evaluators, proving its capability to generate rational cognitive priors.