# Understanding and Visualizing Deep Visual Saliency Models

arXiv: https://arxiv.org/abs/1903.02501 · HTML: https://mlsanity.com/paper/1903.02501/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Mar 2019.

## Built on by (most cited first, 13 of 13)

- [Aerial Vision-and-Dialog Navigation](https://mlsanity.com/paper/2205.12219/) (2022)
- [Towards Deeper Understanding of Camouflaged Object Detection](https://mlsanity.com/paper/2205.11333/) (2022)
- [Understanding Visual Saliency in Mobile User Interfaces](https://mlsanity.com/paper/2101.09176/) (2021)
- [Recent Advances in Leveraging Human Guidance for Sequential Decision-Making Tasks](https://mlsanity.com/paper/2107.05825/) (2021)
- [UniAR: A Unified model for predicting human Attention and Responses on visual content](https://mlsanity.com/paper/2312.10175/) (2023)
- [A Saliency-Guided Street View Image Inpainting Framework for Efficient Last-Meters Wayfinding](https://mlsanity.com/paper/2205.06934/) (2022)
- [What Do Deep Saliency Models Learn about Visual Attention?](https://mlsanity.com/paper/2310.09679/) (2023)
- [How Does Audio Influence Visual Attention in Omnidirectional Videos? Database and Model](https://mlsanity.com/paper/2408.05411/) (2024)
- [Predicting Visual Attention in Graphic Design Documents](https://mlsanity.com/paper/2407.02439/) (2024)
- [Horizontal-to-Vertical Video Conversion](https://mlsanity.com/paper/2101.04051/) (2021)
- [Audio-Visual Collaborative Representation Learning for Dynamic Saliency Prediction](https://mlsanity.com/paper/2109.08371/) (2021)
- [Impacts of the Numbers of Colors and Shapes on Outlier Detection: from Automated to User Evaluation](https://mlsanity.com/paper/2103.06084/) (2021)
- [Deriving Explanation of Deep Visual Saliency Models](https://mlsanity.com/paper/2109.03575/) (2021)

## Abstract (arXiv)

Recently, data-driven deep saliency models have achieved high performance and have outperformed classical saliency models, as demonstrated by results on datasets such as the MIT300 and SALICON. Yet, there remains a large gap between the performance of these models and the inter-human baseline. Some outstanding questions include what have these models learned, how and where they fail, and how they can be improved. This article attempts to answer these questions by analyzing the representations learned by individual neurons located at the intermediate layers of deep saliency models. To this end, we follow the steps of existing deep saliency models, that is borrowing a pre-trained model of object recognition to encode the visual features and learning a decoder to infer the saliency. We consider two cases when the encoder is used as a fixed feature extractor and when it is fine-tuned, and compare the inner representations of the network. To study how the learned representations depend on the task, we fine-tune the same network using the same image set but for two different tasks: saliency prediction versus scene classification. Our analyses reveal that: 1) some visual regions (e.g. head, text, symbol, vehicle) are already encoded within various layers of the network pre-trained for object recognition, 2) using modern datasets, we find that fine-tuning pre-trained models for saliency prediction makes them favor some categories (e.g. head) over some others (e.g. text), 3) although deep models of saliency outperform classical models on natural images, the converse is true for synthetic stimuli (e.g. pop-out search arrays), an evidence of significant difference between human and data-driven saliency models, and 4) we confirm that, after-fine tuning, the change in inner-representations is mostly due to the task and not the domain shift in the data.