# Human Visual Attention Prediction Boosts Learning & Performance of Autonomous Driving Agents

arXiv: https://arxiv.org/abs/1909.05003 · HTML: https://mlsanity.com/paper/1909.05003/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Sep 2019.

## Built on by (most cited first, 7 of 7)

- [Safety at Scale: A Comprehensive Survey of Large Model and Agent Safety](https://mlsanity.com/paper/2502.05206/) (2025)
- [Visual Attention Prediction Improves Performance of Autonomous Drone Racing Agents](https://mlsanity.com/paper/2201.02569/) (2022)
- [Motor Focus: Fast Ego-Motion Prediction for Assistive Visual Navigation](https://mlsanity.com/paper/2404.17031/) (2024)
- [Where and What: Driver Attention-based Object Detection](https://mlsanity.com/paper/2204.12150/) (2022)
- [Guiding Attention in End-to-End Driving Models](https://mlsanity.com/paper/2405.00242/) (2024)
- [Conservative Estimation of Perception Relevance of Dynamic Objects for Safe Trajectories in Automotive Scenarios](https://mlsanity.com/paper/2307.10873/) (2023)
- [FocusDrive: Reasoning with Visual Focus for Autonomous Driving](https://mlsanity.com/paper/2609.33190/) (2026)

## Abstract (arXiv)

Autonomous driving is a multi-task problem requiring a deep understanding of the visual environment. End-to-end autonomous systems have attracted increasing interest as a method of learning to drive without exhaustively programming behaviours for different driving scenarios. When humans drive, they rely on a finely tuned sensory system which enables them to quickly acquire the information they need while filtering unnecessary details. This ability to identify task-specific high-interest regions within an image could be beneficial to autonomous driving agents and machine learning systems in general. To create a system capable of imitating human gaze patterns and visual attention, we collect eye movement data from human drivers in a virtual reality environment. We use this data to train deep neural networks predicting where humans are most likely to look when driving. We then use the outputs of this trained network to selectively mask driving images using a variety of masking techniques. Finally, autonomous driving agents are trained using these masked images as input. Upon comparison, we found that a dual-branch architecture which processes both raw and attention-masked images substantially outperforms all other models, reducing error in control signal predictions by 25.5% compared to a standard end-to-end model trained only on raw images.