# Looking at the right stuff: Guided semantic-gaze for autonomous driving

arXiv: https://arxiv.org/abs/1911.10455 · HTML: https://mlsanity.com/paper/1911.10455/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Nov 2019.

## Built on by (most cited first, 30 of 32)

- [MobileStereoNet: Towards Lightweight Deep Networks for Stereo Matching](https://mlsanity.com/paper/2108.09770/) (2021)
- [Explainable Artificial Intelligence for Autonomous Driving: A Comprehensive Overview and Field Guide for Future Research Directions](https://mlsanity.com/paper/2112.11561/) (2021)
- [MEDIRL: Predicting the Visual Attention of Drivers via Maximum Entropy Deep Inverse Reinforcement Learning](https://mlsanity.com/paper/1912.07773/) (2019)
- [Unsupervised Self-Driving Attention Prediction via Uncertainty Mining and Knowledge Embedding](https://mlsanity.com/paper/2303.09706/) (2023)
- [Salient Sign Detection In Safe Autonomous Driving: AI Which Reasons Over Full Visual Context](https://mlsanity.com/paper/2301.05804/) (2023)
- [On Salience-Sensitive Sign Classification in Autonomous Vehicle Path Planning: Experimental Explorations with a Novel Dataset](https://mlsanity.com/paper/2112.00942/) (2021)
- [RiskBench: A Scenario-based Benchmark for Risk Identification](https://mlsanity.com/paper/2312.01659/) (2023)
- [Robust Traffic Light Detection Using Salience-Sensitive Loss: Computational Framework and Evaluations](https://mlsanity.com/paper/2305.04516/) (2023)
- [Where, What, Why: Towards Explainable Driver Attention Prediction](https://mlsanity.com/paper/2506.23088/) (2025)
- [Behavioral Research and Practical Models of Drivers' Attention](https://mlsanity.com/paper/2104.05677/) (2021)
- [DROID: Driver-centric Risk Object Identification](https://mlsanity.com/paper/2106.13201/) (2021)
- [Trends, Applications, and Challenges in Human Attention Modelling](https://mlsanity.com/paper/2402.18673/) (2024)
- [CoCAtt: A Cognitive-Conditioned Driver Attention Dataset](https://mlsanity.com/paper/2111.10014/) (2021)
- [Modeling Human Gaze Behavior with Diffusion Models for Unified Scanpath Prediction](https://mlsanity.com/paper/2507.23021/) (2025)
- [HiMODE: A Hybrid Monocular Omnidirectional Depth Estimation Model](https://mlsanity.com/paper/2204.05007/) (2022)
- [Improving Generalization of Transfer Learning Across Domains Using Spatio-Temporal Features in Autonomous Driving](https://mlsanity.com/paper/2103.08116/) (2021)
- [SCOUT+: Towards Practical Task-Driven Drivers' Gaze Prediction](https://mlsanity.com/paper/2404.08756/) (2024)
- [Understanding and Modeling the Effects of Task and Context on Drivers' Gaze Allocation](https://mlsanity.com/paper/2310.09275/) (2023)
- [Reason induced visual attention for explainable autonomous driving](https://mlsanity.com/paper/2110.07380/) (2021)
- [Potential Field as Scene Affordance for Behavior Change-Based Visual Risk Object Identification](https://mlsanity.com/paper/2409.15846/) (2024)
- [Where and What: Driver Attention-based Object Detection](https://mlsanity.com/paper/2204.12150/) (2022)
- [CUEING: a lightweight model to Capture hUman attEntion In driviNG](https://mlsanity.com/paper/2305.15710/) (2023)
- [SalFAU-Net: Saliency Fusion Attention U-Net for Salient Object Detection](https://mlsanity.com/paper/2405.02906/) (2024)
- [EraW-Net: Enhance-Refine-Align W-Net for Scene-Associated Driver Attention Estimation](https://mlsanity.com/paper/2408.08570/) (2024)
- [DriverGaze360: OmniDirectional Driver Attention with Object-Level Guidance](https://mlsanity.com/paper/2512.14266/) (2025)
- [CoCAtt: A Cognitive-Conditioned Driver Attention Dataset (Supplementary Material)](https://mlsanity.com/paper/2207.04028/) (2022)
- [Causal Representation-Based Domain Generalization on Gaze Estimation](https://mlsanity.com/paper/2408.16964/) (2024)
- [On-Road Object Importance Estimation: A New Dataset and A Model with Multi-Fold Top-Down Guidance](https://mlsanity.com/paper/2411.17152/) (2024)
- [Towards Robust Unsupervised Attention Prediction in Autonomous Driving](https://mlsanity.com/paper/2501.15045/) (2025)
- [GazeD: Context-Aware Diffusion for Accurate 3D Gaze Estimation](https://mlsanity.com/paper/2601.12948/) (2026)

## Abstract (arXiv)

In recent years, predicting driver's focus of attention has been a very active area of research in the autonomous driving community. Unfortunately, existing state-of-the-art techniques achieve this by relying only on human gaze information, thereby ignoring scene semantics. We propose a novel Semantics Augmented GazE (SAGE) detection approach that captures driving specific contextual information, in addition to the raw gaze. Such a combined attention mechanism serves as a powerful tool to focus on the relevant regions in an image frame in order to make driving both safe and efficient. Using this, we design a complete saliency prediction framework - SAGE-Net, which modifies the initial prediction from SAGE by taking into account vital aspects such as distance to objects (depth), ego vehicle speed, and pedestrian crossing intent. Exhaustive experiments conducted through four popular saliency algorithms show that on mathbf49/56 (87.5%) cases - considering both the overall dataset and crucial driving scenarios, SAGE outperforms existing techniques without any additional computational overhead during the training process. The augmented dataset along with the relevant code are available as part of the supplementary material.