# Gazeformer: Scalable, Effective and Fast Prediction of Goal-Directed Human Attention

arXiv: https://arxiv.org/abs/2303.15274 · HTML: https://mlsanity.com/paper/2303.15274/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 52 references that are in the map.
- First posted on arXiv 27 Mar 2023.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding](https://mlsanity.com/paper/2104.12763/) (2021)
- [Text Summarization with Pretrained Encoders](https://mlsanity.com/paper/1908.08345/) (2019)
- [What do different evaluation metrics tell us about saliency models?](https://mlsanity.com/paper/1604.03605/) (2016)
- [TubeDETR: Spatio-Temporal Video Grounding with Transformers](https://mlsanity.com/paper/2203.16434/) (2022)
- [Deep Gaze I: Boosting Saliency Prediction with Feature Maps Trained on ImageNet](https://mlsanity.com/paper/1411.1045/) (2014)
- [Predicting Goal-directed Human Attention Using Inverse Reinforcement Learning](https://mlsanity.com/paper/2005.14310/) (2020)
- [DeepFix: A Fully Convolutional Neural Network for predicting Human Eye Fixations](https://mlsanity.com/paper/1510.02927/) (2015)
- [Zero-Shot Object Detection: Learning to Simultaneously Recognize and Localize Novel Concepts](https://mlsanity.com/paper/1803.06049/) (2018)
- [Finding any Waldo: zero-shot invariant and efficient visual search](https://mlsanity.com/paper/1807.10587/) (2018)
- [Target-absent Human Attention](https://mlsanity.com/paper/2207.01166/) (2022)
- [Gaze-based dual resolution deep imitation learning for high-precision dexterous robot manipulation](https://mlsanity.com/paper/2102.01295/) (2021)
- [Predicting Goal-directed Attention Control Using Inverse-Reinforcement Learning](https://mlsanity.com/paper/2001.11921/) (2020)

## Built on by (most cited first, 30 of 38)

- [Learning from Observer Gaze:Zero-Shot Attention Prediction Oriented by Human-Object Interaction Recognition](https://mlsanity.com/paper/2405.09931/) (2024)
- [GazeXplain: Learning to Predict Natural Language Explanations of Visual Scanpaths](https://mlsanity.com/paper/2408.02788/) (2024)
- [Beyond Average: Individualized Visual Scanpath Prediction](https://mlsanity.com/paper/2404.12235/) (2024)
- [Trends, Applications, and Challenges in Human Attention Modelling](https://mlsanity.com/paper/2402.18673/) (2024)
- [EyeFormer: Predicting Personalized Scanpaths with Transformer-Guided Reinforcement Learning](https://mlsanity.com/paper/2404.10163/) (2024)
- [UniAR: A Unified model for predicting human Attention and Responses on visual content](https://mlsanity.com/paper/2312.10175/) (2023)
- [Modeling Human Gaze Behavior with Diffusion Models for Unified Scanpath Prediction](https://mlsanity.com/paper/2507.23021/) (2025)
- [OAT: Object-Level Attention Transformer for Gaze Scanpath Prediction](https://mlsanity.com/paper/2407.13335/) (2024)
- [Look Hear: Gaze Prediction for Speech-directed Human Attention](https://mlsanity.com/paper/2407.19605/) (2024)
- [Few-shot Personalized Scanpath Prediction](https://mlsanity.com/paper/2504.05499/) (2025)
- [DiffEye: Diffusion-Based Continuous Eye-Tracking Data Generation Conditioned on Natural Images](https://mlsanity.com/paper/2509.16767/) (2025)
- [Learning User Embeddings from Human Gaze for Personalised Saliency Prediction](https://mlsanity.com/paper/2403.13653/) (2024)
- [Diffusion-Refined VQA Annotations for Semi-Supervised Gaze Following](https://mlsanity.com/paper/2406.02774/) (2024)
- [TPP-Gaze: Modelling Gaze Dynamics in Space and Time with Neural Temporal Point Processes](https://mlsanity.com/paper/2410.23409/) (2024)
- [Measuring and Predicting Where and When Pathologists Focus their Visual Attention while Grading Whole Slide Images of Cancer](https://mlsanity.com/paper/2508.01668/) (2025)
- [Gazing at Rewards: Eye Movements as a Lens into Human and AI Decision-Making in Hybrid Visual Foraging](https://mlsanity.com/paper/2411.09176/) (2024)
- [Human Scanpath Prediction in Target-Present Visual Search with Semantic-Foveal Bayesian Attention](https://mlsanity.com/paper/2507.18503/) (2025)
- [A World Model of Radiologist Reading for Medical Image Representation Learning](https://mlsanity.com/paper/2605.23992/) (2026)
- [Driver Attention Tracking and Analysis](https://mlsanity.com/paper/2404.07122/) (2024)
- [GazeSearch: Radiology Findings Search Benchmark](https://mlsanity.com/paper/2411.05780/) (2024)
- [Unified Attention Modeling for Efficient Free-Viewing and Visual Search via Shared Representations](https://mlsanity.com/paper/2506.02764/) (2025)
- [DeepGaze3.5-VL: Modeling Scanpaths via Autoregressive Token Prediction](https://mlsanity.com/paper/2607.02083/) (2026)
- [Pathformer3D: A 3D Scanpath Transformer for 360deg Images](https://mlsanity.com/paper/2407.10563/) (2024)
- [SIGN: A Statistically-Informed Gaze Network for Gaze Time Prediction](https://mlsanity.com/paper/2501.17422/) (2025)
- [A Comparative Study of Scanpath Models in Graph-Based Visualization](https://mlsanity.com/paper/2503.24160/) (2025)
- [EgoCampus: Egocentric Pedestrian Eye Gaze Model and Dataset](https://mlsanity.com/paper/2512.07668/) (2025)
- [Infinite Gaze Generation for Videos with Autoregressive Diffusion](https://mlsanity.com/paper/2603.24938/) (2026)
- [EVA: Bridging Performance and Human Alignment in Hard-Attention Vision Models for Image Classification](https://mlsanity.com/paper/2603.27340/) (2026)
- [Cost-Efficient Multi-Scale Fovea for Semantic-Based Visual Search Attention](https://mlsanity.com/paper/2604.03836/) (2026)
- [Self-supervised pretraining for an iterative image size agnostic vision transformer](https://mlsanity.com/paper/2604.20392/) (2026)

## Abstract (arXiv)

Predicting human gaze is important in Human-Computer Interaction (HCI). However, to practically serve HCI applications, gaze prediction models must be scalable, fast, and accurate in their spatial and temporal gaze predictions. Recent scanpath prediction models focus on goal-directed attention (search). Such models are limited in their application due to a common approach relying on trained target detectors for all possible objects, and the availability of human gaze data for their training (both not scalable). In response, we pose a new task called ZeroGaze, a new variant of zero-shot learning where gaze is predicted for never-before-searched objects, and we develop a novel model, Gazeformer, to solve the ZeroGaze problem. In contrast to existing methods using object detector modules, Gazeformer encodes the target using a natural language model, thus leveraging semantic similarities in scanpath prediction. We use a transformer-based encoder-decoder architecture because transformers are particularly useful for generating contextual representations. Gazeformer surpasses other models by a large margin on the ZeroGaze setting. It also outperforms existing target-detection models on standard gaze prediction for both target-present and target-absent search tasks. In addition to its improved performance, Gazeformer is more than five times faster than the state-of-the-art target-present visual search model.