# EyeFormer: Predicting Personalized Scanpaths with Transformer-Guided Reinforcement Learning

arXiv: https://arxiv.org/abs/2404.10163 · HTML: https://mlsanity.com/paper/2404.10163/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 24 of its 72 references that are in the map.
- First posted on arXiv 15 Apr 2024.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Sequence Level Training with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06732/) (2015)
- [Self-critical Sequence Training for Image Captioning](https://mlsanity.com/paper/1612.00563/) (2016)
- [When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations](https://mlsanity.com/paper/2106.01548/) (2021)
- [Discretizing Continuous Action Space for On-Policy Optimization](https://mlsanity.com/paper/1901.10500/) (2019)
- [Predicting Goal-directed Human Attention Using Inverse Reinforcement Learning](https://mlsanity.com/paper/2005.14310/) (2020)
- [Gazeformer: Scalable, Effective and Fast Prediction of Goal-Directed Human Attention](https://mlsanity.com/paper/2303.15274/) (2023)
- [Predicting Head Movement in Panoramic Video: A Deep Reinforcement Learning Approach](https://mlsanity.com/paper/1710.10755/) (2017)
- [PathGAN: Visual Scanpath Prediction with Generative Adversarial Networks](https://mlsanity.com/paper/1809.00567/) (2018)
- [Predicting Visual Importance Across Graphic Design Types](https://mlsanity.com/paper/2008.02912/) (2020)
- [SaltiNet: Scan-path Prediction on 360 Degree Images using Saliency Volumes](https://mlsanity.com/paper/1707.03123/) (2017)
- [CLIP4IDC: CLIP for Image Difference Captioning](https://mlsanity.com/paper/2206.00629/) (2022)
- [ScanGAN360: A Generative Model of Realistic Scanpaths for 360circ Images](https://mlsanity.com/paper/2103.13922/) (2021)
- [Understanding Visual Saliency in Mobile User Interfaces](https://mlsanity.com/paper/2101.09176/) (2021)
- [GRIDS: Interactive Layout Design with Integer Programming](https://mlsanity.com/paper/2001.02921/) (2020)
- [Behind the Machine's Gaze: Neural Networks with Biologically-inspired Constraints Exhibit Human-like Visual Attention](https://mlsanity.com/paper/2204.09093/) (2022)
- [ORC Layout: Adaptive GUI Layout with OR-Constraints](https://mlsanity.com/paper/1912.07827/) (2019)
- [ORCSolver: An Efficient Solver for Adaptive GUI Layout with OR-Constraints](https://mlsanity.com/paper/2002.09925/) (2020)
- [ReverseORC: Reverse Engineering of Resizable User Interface Layouts with OR-Constraints](https://mlsanity.com/paper/2202.11523/) (2022)
- [Learning to predict where to look in interactive environments using deep recurrent q-learning](https://mlsanity.com/paper/1612.05753/) (2016)
- [PiTL: Cross-modal Retrieval with Weakly-supervised Vision-language Pre-training via Prompting](https://mlsanity.com/paper/2307.07341/) (2023)
- [A Probabilistic Time-Evolving Approach to Scanpath Prediction](https://mlsanity.com/paper/2204.09404/) (2022)
- [UEyes: An Eye-Tracking Dataset across User Interface Types](https://mlsanity.com/paper/2402.05202/) (2024)

## Built on by (most cited first, 8 of 8)

- [Few-shot Personalized Scanpath Prediction](https://mlsanity.com/paper/2504.05499/) (2025)
- [AgentForge: A Flexible Low-Code Platform for Reinforcement Learning Agent Design](https://mlsanity.com/paper/2410.19528/) (2024)
- [ControlGUI: Guiding Generative GUI Exploration through Perceptual Visual Flow](https://mlsanity.com/paper/2502.03330/) (2025)
- [Intelligent support for Human Oversight: Integrating Reinforcement Learning with Gaze Simulation to Personalize Highlighting](https://mlsanity.com/paper/2602.08403/) (2026)
- [Simulation-based Optimization for Augmented Reading](https://mlsanity.com/paper/2602.22735/) (2026)
- [A Resource-Rational Principle for Modeling Visual Attention Control](https://mlsanity.com/paper/2603.02056/) (2026)
- [Self-supervised pretraining for an iterative image size agnostic vision transformer](https://mlsanity.com/paper/2604.20392/) (2026)
- [ST-DiffEye: Diffusion-based Continuous Gaze Generation via Joint Scanpath-Trajectory Modeling](https://mlsanity.com/paper/2606.15486/) (2026)

## Abstract (arXiv)

From a visual perception perspective, modern graphical user interfaces (GUIs) comprise a complex graphics-rich two-dimensional visuospatial arrangement of text, images, and interactive objects such as buttons and menus. While existing models can accurately predict regions and objects that are likely to attract attention ``on average'', so far there is no scanpath model capable of predicting scanpaths for an individual. To close this gap, we introduce EyeFormer, which leverages a Transformer architecture as a policy network to guide a deep reinforcement learning algorithm that controls gaze locations. Our model has the unique capability of producing personalized predictions when given a few user scanpath samples. It can predict full scanpath information, including fixation positions and duration, across individuals and various stimulus types. Additionally, we demonstrate applications in GUI layout optimization driven by our model. Our software and models will be publicly available.