# Actions in the Eye: Dynamic Gaze Datasets and Learnt Saliency Models for Visual Recognition

arXiv: https://arxiv.org/abs/1312.7570 · HTML: https://mlsanity.com/paper/1312.7570/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Dec 2013.

## Built on by (most cited first, 30 of 37)

- [Predicting Goal-directed Human Attention Using Inverse Reinforcement Learning](https://mlsanity.com/paper/2005.14310/) (2020)
- [MEDIRL: Predicting the Visual Attention of Drivers via Maximum Entropy Deep Inverse Reinforcement Learning](https://mlsanity.com/paper/1912.07773/) (2019)
- [Temporal-Spatial Feature Pyramid for Video Saliency Detection](https://mlsanity.com/paper/2105.04213/) (2021)
- [Human Attention in Fine-grained Classification](https://mlsanity.com/paper/2111.01628/) (2021)
- [Spatio-Temporal Self-Attention Network for Video Saliency Prediction](https://mlsanity.com/paper/2108.10696/) (2021)
- [Recent Advances in Leveraging Human Guidance for Sequential Decision-Making Tasks](https://mlsanity.com/paper/2107.05825/) (2021)
- [CaRDiff: Video Salient Object Ranking Chain of Thought Reasoning for Saliency Prediction with Diffusion](https://mlsanity.com/paper/2408.12009/) (2024)
- [SalFoM: Dynamic Saliency Prediction with Video Foundation Models](https://mlsanity.com/paper/2404.03097/) (2024)
- [NTIRE 2026 Challenge on Video Saliency Prediction: Methods and Results](https://mlsanity.com/paper/2604.14816/) (2026)
- [MAAD: A Model and Dataset for "Attended Awareness" in Driving](https://mlsanity.com/paper/2110.08610/) (2021)
- [TinyHD: Efficient Video Saliency Prediction with Heterogeneous Decoders using Hierarchical Maps Distillation](https://mlsanity.com/paper/2301.04619/) (2023)
- [Transformer-based Video Saliency Prediction with High Temporal Dimension Decoding](https://mlsanity.com/paper/2401.07942/) (2024)
- [AIM 2024 Challenge on Video Saliency Prediction: Methods and Results](https://mlsanity.com/paper/2409.14827/) (2024)
- [How Does Audio Influence Visual Attention in Omnidirectional Videos? Database and Model](https://mlsanity.com/paper/2408.05411/) (2024)
- [RadGazeGen: Radiomics and Gaze-guided Medical Image Generation using Diffusion Models](https://mlsanity.com/paper/2410.00307/) (2024)
- [Minimalistic Video Saliency Prediction via Efficient Decoder & Spatio Temporal Action Cues](https://mlsanity.com/paper/2502.00397/) (2025)
- [A Gated Fusion Network for Dynamic Saliency Prediction](https://mlsanity.com/paper/2102.07682/) (2021)
- [Data Limitations for Modeling Top-Down Effects on Drivers' Attention](https://mlsanity.com/paper/2404.08749/) (2024)
- [Where to look at the movies : Analyzing visual attention to understand movie editing](https://mlsanity.com/paper/2102.13378/) (2021)
- [Modeling Long-Term Memory and Temporal Attention Shifts for Video Salient Object Ranking with a New Benchmark](https://mlsanity.com/paper/2203.17257/) (2022)
- [Evaluation of Explanation Methods of AI -- CNNs in Image Classification Tasks with Reference-based and No-reference Metrics](https://mlsanity.com/paper/2212.01222/) (2022)
- [Attention in Reasoning: Dataset, Analysis, and Modeling](https://mlsanity.com/paper/2204.09774/) (2022)
- [NPF-200: A Multi-Modal Eye Fixation Dataset and Method for Non-Photorealistic Videos](https://mlsanity.com/paper/2308.12163/) (2023)
- [ConsistencyDet: A Few-step Denoising Framework for Object Detection Using the Consistency Model](https://mlsanity.com/paper/2404.07773/) (2024)
- [GazeLT: Visual attention-guided long-tailed disease classification in chest radiographs](https://mlsanity.com/paper/2508.09478/) (2025)
- [Exploring deep learning for Event-Based Saliency Prediction with a Transformer-based model](https://mlsanity.com/paper/2605.23790/) (2026)
- [Attend to Anything: Foundation Model for Unified Human Attention Modeling](https://mlsanity.com/paper/2606.03540/) (2026)
- [Deriving Explanation of Deep Visual Saliency Models](https://mlsanity.com/paper/2109.03575/) (2021)
- [ViDaS Video Depth-aware Saliency Network](https://mlsanity.com/paper/2305.11729/) (2023)
- [Artificially Generated Visual Scanpath Improves Multi-label Thoracic Disease Classification in Chest X-Ray Images](https://mlsanity.com/paper/2503.00657/) (2025)

## Abstract (arXiv)

Systems based on bag-of-words models from image features collected at maxima of sparse interest point operators have been used successfully for both computer visual object and action recognition tasks. While the sparse, interest-point based approach to recognition is not inconsistent with visual processing in biological systems that operate in `saccade and fixate' regimes, the methodology and emphasis in the human and the computer vision communities remains sharply distinct. Here, we make three contributions aiming to bridge this gap. First, we complement existing state-of-the art large scale dynamic computer vision annotated datasets like Hollywood-2 and UCF Sports with human eye movements collected under the ecological constraints of the visual action recognition task. To our knowledge these are the first large human eye tracking datasets to be collected and made publicly available for video, vision.imar.ro/eyetracking (497,107 frames, each viewed by 16 subjects), unique in terms of their (a) large scale and computer vision relevance, (b) dynamic, video stimuli, (c) task control, as opposed to free-viewing. Second, we introduce novel sequential consistency and alignment measures, which underline the remarkable stability of patterns of visual search among subjects. Third, we leverage the significant amount of collected data in order to pursue studies and build automatic, end-to-end trainable computer vision systems based on human eye movements. Our studies not only shed light on the differences between computer vision spatio-temporal interest point image sampling strategies and the human fixations, as well as their impact for visual recognition performance, but also demonstrate that human fixations can be accurately predicted, and when used in an end-to-end automatic system, leveraging some of the advanced computer vision practice, can lead to state of the art results.