# DiffAttn: Diffusion-Based Drivers' Visual Attention Prediction with LLM-Enhanced Semantic Reasoning

arXiv: https://arxiv.org/abs/2603.28251 · HTML: https://mlsanity.com/paper/2603.28251/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Chain-of-thought reasoning for driving, one of 17 directions in LLM reasoning.
- Builds on 20 of its 35 references that are in the map.
- First posted on arXiv 30 Mar 2026.

## Placed in

- LLM reasoning / [Chain-of-thought reasoning for driving](https://mlsanity.com/lr/chain-of-thought-reasoning-for-driving/)

## Builds on (most cited first, 20 of 20)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Improved Techniques for Training Score-Based Generative Models](https://mlsanity.com/paper/2006.09011/) (2020)
- [Pyramid Feature Attention Network for Saliency detection](https://mlsanity.com/paper/1903.00179/) (2019)
- [DADA: Driver Attention Prediction in Driving Accident Scenarios](https://mlsanity.com/paper/1912.12148/) (2019)
- [Revisiting Video Saliency: A Large-scale Benchmark and a New Model](https://mlsanity.com/paper/1801.07424/) (2018)
- [Predicting Driver Attention in Critical Situations](https://mlsanity.com/paper/1711.06406/) (2017)
- [TASED-Net: Temporally-Aggregating Spatial Encoder-Decoder Network for Video Saliency Detection](https://mlsanity.com/paper/1908.05786/) (2019)
- [A Deep Multi-Level Network for Saliency Prediction](https://mlsanity.com/paper/1609.01064/) (2016)
- [MEDIRL: Predicting the Visual Attention of Drivers via Maximum Entropy Deep Inverse Reinforcement Learning](https://mlsanity.com/paper/1912.07773/) (2019)
- [Simple vs complex temporal recurrences for video saliency prediction](https://mlsanity.com/paper/1907.01869/) (2019)
- [DeepTake: Prediction of Driver Takeover Behavior using Multimodal Data](https://mlsanity.com/paper/2012.15441/) (2020)
- [FBLNet: FeedBack Loop Network for Driver Attention Prediction](https://mlsanity.com/paper/2212.02096/) (2022)
- [Pre-Trained LLM is a Semantic-Aware and Generalizable Segmentation Booster](https://mlsanity.com/paper/2506.18034/) (2025)
- [SCOUT+: Towards Practical Task-Driven Drivers' Gaze Prediction](https://mlsanity.com/paper/2404.08756/) (2024)
- [SalM2: An Extremely Lightweight Saliency Mamba Model for Real-Time Cognitive Awareness of Driver Attention](https://mlsanity.com/paper/2502.16214/) (2025)
- [Gated Driver Attention Predictor](https://mlsanity.com/paper/2308.02530/) (2023)

## Abstract (arXiv)

Drivers' visual attention provides critical cues for anticipating latent hazards and directly shapes decision-making and control maneuvers, where its absence can compromise traffic safety. To emulate drivers' perception patterns and advance visual attention prediction for intelligent vehicles, we propose DiffAttn, a diffusion-based framework that formulates this task as a conditional diffusion-denoising process, enabling more accurate modeling of drivers' attention. To capture both local and global scene features, we adopt Swin Transformer as encoder and design a decoder that combines a Feature Fusion Pyramid for cross-layer interaction with dense, multi-scale conditional diffusion to jointly enhance denoising learning and model fine-grained local and global scene contexts. Additionally, a large language model (LLM) layer is incorporated to enhance top-down semantic reasoning and improve sensitivity to safety-critical cues. Extensive experiments on four public datasets demonstrate that DiffAttn achieves state-of-the-art (SoTA) performance, surpassing most video-based, top-down-feature-driven, and LLM-enhanced baselines. Our framework further supports interpretable driver-centric scene understanding and has the potential to improve in-cabin human-machine interaction, risk perception, and drivers' state measurement in intelligent vehicles.