# Data Limitations for Modeling Top-Down Effects on Drivers' Attention

arXiv: https://arxiv.org/abs/2404.08749 · HTML: https://mlsanity.com/paper/2404.08749/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 38 references that are in the map.
- First posted on arXiv 12 Apr 2024.

## Builds on (most cited first, 12 of 12)

- [ORB-SLAM: a Versatile and Accurate Monocular SLAM System](https://mlsanity.com/paper/1502.00956/) (2015)
- [What do different evaluation metrics tell us about saliency models?](https://mlsanity.com/paper/1604.03605/) (2016)
- [End-to-end Learning of Driving Models from Large-scale Video Datasets](https://mlsanity.com/paper/1612.01079/) (2016)
- [Predicting the Driver's Focus of Attention: the DR(eye)VE Project](https://mlsanity.com/paper/1705.03854/) (2017)
- [DADA: Driver Attention Prediction in Driving Accident Scenarios](https://mlsanity.com/paper/1912.12148/) (2019)
- [Predicting Driver Attention in Critical Situations](https://mlsanity.com/paper/1711.06406/) (2017)
- [ViNet: Pushing the limits of Visual Modality for Audio-Visual Saliency Prediction](https://mlsanity.com/paper/2012.06170/) (2020)
- [Actions in the Eye: Dynamic Gaze Datasets and Learnt Saliency Models for Visual Recognition](https://mlsanity.com/paper/1312.7570/) (2013)
- [MEDIRL: Predicting the Visual Attention of Drivers via Maximum Entropy Deep Inverse Reinforcement Learning](https://mlsanity.com/paper/1912.07773/) (2019)
- [Behavioral Research and Practical Models of Drivers' Attention](https://mlsanity.com/paper/2104.05677/) (2021)
- [MAAD: A Model and Dataset for "Attended Awareness" in Driving](https://mlsanity.com/paper/2110.08610/) (2021)
- [Understanding and Modeling the Effects of Task and Context on Drivers' Gaze Allocation](https://mlsanity.com/paper/2310.09275/) (2023)

## Built on by (most cited first, 3 of 3)

- [A Survey on Diffusion Models for Time Series and Spatio-Temporal Data](https://mlsanity.com/paper/2404.18886/) (2024)
- [Where, What, Why: Towards Explainable Driver Attention Prediction](https://mlsanity.com/paper/2506.23088/) (2025)
- [Beyond Scanpaths: Graph-Based Gaze Simulation in Dynamic Scenes](https://mlsanity.com/paper/2603.28319/) (2026)

## Abstract (arXiv)

Driving is a visuomotor task, i.e., there is a connection between what drivers see and what they do. While some models of drivers' gaze account for top-down effects of drivers' actions, the majority learn only bottom-up correlations between human gaze and driving footage. The crux of the problem is lack of public data with annotations that could be used to train top-down models and evaluate how well models of any kind capture effects of task on attention. As a result, top-down models are trained and evaluated on private data and public benchmarks measure only the overall fit to human data. In this paper, we focus on data limitations by examining four large-scale public datasets, DR(eye)VE, BDD-A, MAAD, and LBW, used to train and evaluate algorithms for drivers' gaze prediction. We define a set of driving tasks (lateral and longitudinal maneuvers) and context elements (intersections and right-of-way) known to affect drivers' attention, augment the datasets with annotations based on the said definitions, and analyze the characteristics of data recording and processing pipelines w.r.t. capturing what the drivers see and do. In sum, the contributions of this work are: 1) quantifying biases of the public datasets, 2) examining performance of the SOTA bottom-up models on subsets of the data involving non-trivial drivers' actions, 3) linking shortcomings of the bottom-up models to data limitations, and 4) recommendations for future data collection and processing. The new annotations and code for reproducing the results is available at https://github.com/ykotseruba/SCOUT.