# Improving saliency models' predictions of the next fixation with humans' intrinsic cost of gaze shifts

arXiv: https://arxiv.org/abs/2207.04250 · HTML: https://mlsanity.com/paper/2207.04250/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 65 references that are in the map.
- First posted on arXiv 9 Jul 2022.

## Builds on (most cited first, 15 of 15)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [Recurrent Models of Visual Attention](https://mlsanity.com/paper/1406.6247/) (2014)
- [What do different evaluation metrics tell us about saliency models?](https://mlsanity.com/paper/1604.03605/) (2016)
- [Predicting Human Eye Fixations via an LSTM-based Saliency Attentive Model](https://mlsanity.com/paper/1611.09571/) (2016)
- [Unified Image and Video Saliency Modeling](https://mlsanity.com/paper/2003.05477/) (2020)
- [Revisiting Video Saliency: A Large-scale Benchmark and a New Model](https://mlsanity.com/paper/1801.07424/) (2018)
- [EML-NET:An Expandable Multi-Layer NETwork for Saliency Prediction](https://mlsanity.com/paper/1805.01047/) (2018)
- [DeepFix: A Fully Convolutional Neural Network for predicting Human Eye Fixations](https://mlsanity.com/paper/1510.02927/) (2015)
- [Shallow and Deep Convolutional Networks for Saliency Prediction](https://mlsanity.com/paper/1603.00845/) (2016)
- [PathGAN: Visual Scanpath Prediction with Generative Adversarial Networks](https://mlsanity.com/paper/1809.00567/) (2018)
- [SaltiNet: Scan-path Prediction on 360 Degree Images using Saliency Volumes](https://mlsanity.com/paper/1707.03123/) (2017)
- [State-of-the-Art in Human Scanpath Prediction](https://mlsanity.com/paper/2102.12239/) (2021)
- [Saliency Prediction in the Deep Learning Era: Successes, Limitations, and Future Challenges](https://mlsanity.com/paper/1810.03716/) (2018)
- [Saliency-based Sequential Image Attention with Multiset Prediction](https://mlsanity.com/paper/1711.05165/) (2017)

## Built on by (most cited first, 4 of 4)

- [Unified Dynamic Scanpath Predictors Outperform Individually Trained Neural Models](https://mlsanity.com/paper/2405.02929/) (2024)
- [Raising the Ceiling: Better Empirical Fixation Densities for Saliency Benchmarking](https://mlsanity.com/paper/2605.03885/) (2026)
- [Object segmentation in the wild with foundation models: application to vision assisted neuro-prostheses for upper limbs](https://mlsanity.com/paper/2507.18517/) (2025)
- [DIJIT: A Robotic Head for an Active Observer](https://mlsanity.com/paper/2512.07998/) (2025)

## Abstract (arXiv)

The human prioritization of image regions can be modeled in a time invariant fashion with saliency maps or sequentially with scanpath models. However, while both types of models have steadily improved on several benchmarks and datasets, there is still a considerable gap in predicting human gaze. Here, we leverage two recent developments to reduce this gap: theoretical analyses establishing a principled framework for predicting the next gaze target and the empirical measurement of the human cost for gaze switches independently of image content. We introduce an algorithm in the framework of sequential decision making, which converts any static saliency map into a sequence of dynamic history-dependent value maps, which are recomputed after each gaze shift. These maps are based on 1) a saliency map provided by an arbitrary saliency model, 2) the recently measured human cost function quantifying preferences in magnitude and direction of eye movements, and 3) a sequential exploration bonus, which changes with each subsequent gaze shift. The parameters of the spatial extent and temporal decay of this exploration bonus are estimated from human gaze data. The relative contributions of these three components were optimized on the MIT1003 dataset for the NSS score and are sufficient to significantly outperform predictions of the next gaze target on NSS and AUC scores for five state of the art saliency models on three image data sets. Thus, we provide an implementation of human gaze preferences, which can be used to improve arbitrary saliency models' predictions of humans' next gaze targets.