# Modeling Human Gaze Behavior with Diffusion Models for Unified Scanpath Prediction

arXiv: https://arxiv.org/abs/2507.23021 · HTML: https://mlsanity.com/paper/2507.23021/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Video generation, one of 14 directions in Diffusion models.
- Builds on 42 of its 74 references that are in the map.
- First posted on arXiv 30 Jul 2025.

## Placed in

- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)

## Builds on (most cited first, 30 of 42)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [DiffWave: A Versatile Diffusion Model for Audio Synthesis](https://mlsanity.com/paper/2009.09761/) (2020)
- [Diffusion-LM Improves Controllable Text Generation](https://mlsanity.com/paper/2205.14217/) (2022)
- [Diffusion Models in Vision: A Survey](https://mlsanity.com/paper/2209.04747/) (2022)
- [Vision Transformers Need Registers](https://mlsanity.com/paper/2309.16588/) (2023)
- [MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model](https://mlsanity.com/paper/2208.15001/) (2022)
- [DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models](https://mlsanity.com/paper/2210.08933/) (2022)
- [A Survey on Diffusion Models for Time Series and Spatio-Temporal Data](https://mlsanity.com/paper/2404.18886/) (2024)
- [DiffTraj: Generating GPS Trajectory with Diffusion Probabilistic Model](https://mlsanity.com/paper/2304.11582/) (2023)
- [Deep Gaze I: Boosting Saliency Prediction with Feature Maps Trained on ImageNet](https://mlsanity.com/paper/1411.1045/) (2014)
- [On the Constrained Time-Series Generation Problem](https://mlsanity.com/paper/2307.01717/) (2023)
- [Predicting Goal-directed Human Attention Using Inverse Reinforcement Learning](https://mlsanity.com/paper/2005.14310/) (2020)
- [ControlTraj: Controllable Trajectory Generation with Topology-Constrained Diffusion Model](https://mlsanity.com/paper/2404.15380/) (2024)
- [Gazeformer: Scalable, Effective and Fast Prediction of Goal-Directed Human Attention](https://mlsanity.com/paper/2303.15274/) (2023)
- [Time Series Diffusion in the Frequency Domain](https://mlsanity.com/paper/2402.05933/) (2024)
- [PathGAN: Visual Scanpath Prediction with Generative Adversarial Networks](https://mlsanity.com/paper/1809.00567/) (2018)
- [Looking at the right stuff: Guided semantic-gaze for autonomous driving](https://mlsanity.com/paper/1911.10455/) (2019)
- [Perception and Navigation in Autonomous Systems in the Era of Learning: A Survey](https://mlsanity.com/paper/2001.02319/) (2020)
- [Diff-RNTraj: A Structure-aware Diffusion Model for Road Network-constrained Trajectory Generation](https://mlsanity.com/paper/2402.07369/) (2024)
- [Regular Time-series Generation using SGM](https://mlsanity.com/paper/2301.08518/) (2023)
- [State-of-the-Art in Human Scanpath Prediction](https://mlsanity.com/paper/2102.12239/) (2021)

## Built on by (most cited first, 7 of 7)

- [DeepGaze3.5-VL: Modeling Scanpaths via Autoregressive Token Prediction](https://mlsanity.com/paper/2607.02083/) (2026)
- [EgoCampus: Egocentric Pedestrian Eye Gaze Model and Dataset](https://mlsanity.com/paper/2512.07668/) (2025)
- [EVA: Bridging Performance and Human Alignment in Hard-Attention Vision Models for Image Classification](https://mlsanity.com/paper/2603.27340/) (2026)
- [ST-DiffEye: Diffusion-based Continuous Gaze Generation via Joint Scanpath-Trajectory Modeling](https://mlsanity.com/paper/2606.15486/) (2026)
- [Attention Capture Is Not Detection: A Two-Stage Account of How Humans Miss Localized AI Image Edits](https://mlsanity.com/paper/2608.13865/) (2026)
- [Diffusion models for eye-gaze trajectory generation using position and velocity representations](https://mlsanity.com/paper/2609.05522/) (2026)
- [Learning to Refer from Estimated Listener Gaze](https://mlsanity.com/paper/2609.14207/) (2026)

## Abstract (arXiv)

Predicting human gaze scanpaths is crucial for understanding visual attention, with applications in human-computer interaction, autonomous systems, and cognitive robotics. While deep learning models have advanced scanpath prediction, most existing approaches generate averaged behaviors, failing to capture the variability of human visual exploration. In this work, we present ScanDiff, a novel architecture that combines diffusion models with Vision Transformers to generate diverse and realistic scanpaths. Our method explicitly models scanpath variability by leveraging the stochastic nature of diffusion models, producing a wide range of plausible gaze trajectories. Additionally, we introduce textual conditioning to enable task-driven scanpath generation, allowing the model to adapt to different visual search objectives. Experiments on benchmark datasets show that ScanDiff surpasses state-of-the-art methods in both free-viewing and task-driven scenarios, producing more diverse and accurate scanpaths. These results highlight its ability to better capture the complexity of human visual behavior, pushing forward gaze prediction research. Source code and models are publicly available at https://aimagelab.github.io/ScanDiff.