# Beyond Scanpaths: Graph-Based Gaze Simulation in Dynamic Scenes

arXiv: https://arxiv.org/abs/2603.28319 · HTML: https://mlsanity.com/paper/2603.28319/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 149 references that are in the map.
- First posted on arXiv 30 Mar 2026.

## Builds on (most cited first, 15 of 15)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges](https://mlsanity.com/paper/2104.13478/) (2021)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [Learning Mesh-Based Simulation with Graph Networks](https://mlsanity.com/paper/2010.03409/) (2020)
- [SalGAN: Visual Saliency Prediction with Generative Adversarial Networks](https://mlsanity.com/paper/1701.01081/) (2017)
- [Deep Gaze I: Boosting Saliency Prediction with Feature Maps Trained on ImageNet](https://mlsanity.com/paper/1411.1045/) (2014)
- [Eagle: Large-Scale Learning of Turbulent Fluid Dynamics with Mesh Transformers](https://mlsanity.com/paper/2302.10803/) (2023)
- [YOLOPv2: Better, Faster, Stronger for Panoptic Driving Perception](https://mlsanity.com/paper/2208.11434/) (2022)
- [Heterogeneous Edge-Enhanced Graph Attention Network For Multi-Agent Trajectory Prediction](https://mlsanity.com/paper/2106.07161/) (2021)
- [Data Limitations for Modeling Top-Down Effects on Drivers' Attention](https://mlsanity.com/paper/2404.08749/) (2024)
- [Learning Physical Simulation with Message Passing Transformer](https://mlsanity.com/paper/2406.06060/) (2024)
- [DiffGaze: A Diffusion Model for Continuous Gaze Sequence Generation on 360deg Images](https://mlsanity.com/paper/2403.17477/) (2024)

## Abstract (arXiv)

Accurately modelling human attention is essential for numerous computer vision applications, particularly in the domain of automotive safety. Existing methods typically collapse gaze into saliency maps or scanpaths, treating gaze dynamics only implicitly. We instead formulate gaze modelling as an autoregressive dynamical system and explicitly unroll raw gaze trajectories over time, conditioned on both gaze history and the evolving environment. Driving scenes are represented as gaze-centric graphs processed by the Affinity Relation Transformer (ART), a heterogeneous graph transformer that models interactions between driver gaze, traffic objects, and road structure. We further introduce the Object Density Network (ODN) to predict next-step gaze distributions, capturing the stochastic and object-centric nature of attentional shifts in complex environments. We also release Focus100, a new dataset of raw gaze data from 30 participants viewing egocentric driving footage. Trained directly on raw gaze, without fixation filtering, our unified approach produces more natural gaze trajectories, scanpath dynamics, and saliency maps than existing attention models, offering valuable insights for the temporal modelling of human attention in dynamic environments.