mlsanityData to 7 Oct 2026

Improving saliency models' predictions of the next fixation with humans' intrinsic cost of gaze shifts

arXiv 2207.04250 ↗First posted 9 Jul 2022cs.CVcited by 4

Not on the map. Why?

Abstract (arXiv)

The human prioritization of image regions can be modeled in a time invariant fashion with saliency maps or sequentially with scanpath models. However, while both types of models have steadily improved on several benchmarks and datasets, there is still a considerable gap in predicting human gaze. Here, we leverage two recent developments to reduce this gap: theoretical analyses establishing a principled framework for predicting the next gaze target and the empirical measurement of the human cost for gaze switches independently of image content. We introduce an algorithm in the framework of sequential decision making, which converts any static saliency map into a sequence of dynamic history-dependent value maps, which are recomputed after each gaze shift. These maps are based on 1) a saliency map provided by an arbitrary saliency model, 2) the recently measured human cost function quantifying preferences in magnitude and direction of eye movements, and 3) a sequential exploration bonus, which changes with each subsequent gaze shift. The parameters of the spatial extent and temporal decay of this exploration bonus are estimated from human gaze data. The relative contributions of these three components were optimized on the MIT1003 dataset for the NSS score and are sufficient to significantly outperform predictions of the next gaze target on NSS and AUC scores for five state of the art saliency models on three image data sets. Thus, we provide an implementation of human gaze preferences, which can be used to improve arbitrary saliency models' predictions of humans' next gaze targets.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 15 of its 65 references that are in the map. First posted on arXiv 9 Jul 2022.

Builds on

15 of 65 references.

References · 15

  1. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  2. 2018Sanity Checks for Saliency Mapscited by 1,149
  3. 2014Recurrent Models of Visual Attentioncited by 344
  4. 2016What do different evaluation metrics tell us about saliency models?cited by 184
  5. 2016Predicting Human Eye Fixations via an LSTM-based Saliency Attentive Modelcited by 91
  6. 2020Unified Image and Video Saliency Modelingcited by 74
  7. 2018Revisiting Video Saliency: A Large-scale Benchmark and a New Modelcited by 66
  8. 2018EML-NET:An Expandable Multi-Layer NETwork for Saliency Predictioncited by 51
  9. 2015DeepFix: A Fully Convolutional Neural Network for predicting Human Eye Fixationscited by 47
  10. 2016Shallow and Deep Convolutional Networks for Saliency Predictioncited by 42
  11. 2018PathGAN: Visual Scanpath Prediction with Generative Adversarial Networkscited by 32
  12. 2017SaltiNet: Scan-path Prediction on 360 Degree Images using Saliency Volumescited by 29
3 more
  1. 2021State-of-the-Art in Human Scanpath Predictioncited by 22
  2. 2018Saliency Prediction in the Deep Learning Era: Successes, Limitations, and Future Challengescited by 7
  3. 2017Saliency-based Sequential Image Attention with Multiset Predictioncited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 1.

4 papers.

  1. 2024Unified Dynamic Scanpath Predictors Outperform Individually Trained Neural Modelscited by 0
  2. 2025Object segmentation in the wild with foundation models: application to vision assisted neuro-prostheses for upper limbscited by 0
  3. 2025DIJIT: A Robotic Head for an Active Observercited by 0
  4. 2026Raising the Ceiling: Better Empirical Fixation Densities for Saliency Benchmarkingcited by 0