# Saliency Prediction in the Deep Learning Era: Successes, Limitations, and Future Challenges

arXiv: https://arxiv.org/abs/1810.03716 · HTML: https://mlsanity.com/paper/1810.03716/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Oct 2018.

## Built on by (most cited first, 8 of 8)

- [Image Cropping on Twitter: Fairness Metrics, their Limitations, and the Importance of Representation, Design, and Agency](https://mlsanity.com/paper/2105.08667/) (2021)
- [Understanding Visual Saliency in Mobile User Interfaces](https://mlsanity.com/paper/2101.09176/) (2021)
- [Improving saliency models' predictions of the next fixation with humans' intrinsic cost of gaze shifts](https://mlsanity.com/paper/2207.04250/) (2022)
- [Where and What: Driver Attention-based Object Detection](https://mlsanity.com/paper/2204.12150/) (2022)
- [Selective Attention-based Modulation for Continual Learning](https://mlsanity.com/paper/2403.20086/) (2024)
- [Visual attention information can be traced on cortical response but not on the retina: evidence from electrophysiological mouse data using natural images as stimuli](https://mlsanity.com/paper/2308.00526/) (2023)
- [A Saccade-inspired Approach to Image Classification using Vision Transformer Attention Maps](https://mlsanity.com/paper/2603.09613/) (2026)
- [Semantic Saliency from Multi-Modal Large Language Model Scene Understanding Maps](https://mlsanity.com/paper/2610.15467/) (2026)

## Abstract (arXiv)

Visual saliency models have enjoyed a big leap in performance in recent years, thanks to advances in deep learning and large scale annotated data. Despite enormous effort and huge breakthroughs, however, models still fall short in reaching human-level accuracy. In this work, I explore the landscape of the field emphasizing on new deep saliency models, benchmarks, and datasets. A large number of image and video saliency models are reviewed and compared over two image benchmarks and two large scale video datasets. Further, I identify factors that contribute to the gap between models and humans and discuss remaining issues that need to be addressed to build the next generation of more powerful saliency models. Some specific questions that are addressed include: in what ways current models fail, how to remedy them, what can be learned from cognitive studies of attention, how explicit saliency judgments relate to fixations, how to conduct fair model comparison, and what are the emerging applications of saliency models.