# Deep Saliency Models : The Quest For The Loss Function

arXiv: https://arxiv.org/abs/1907.02336 · HTML: https://mlsanity.com/paper/1907.02336/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Jul 2019.

## Built on by (most cited first, 8 of 8)

- [Affordance Grounding from Demonstration Video to Target Image](https://mlsanity.com/paper/2303.14644/) (2023)
- [CYBORG: Blending Human Saliency Into the Loss Improves Deep Learning](https://mlsanity.com/paper/2112.00686/) (2021)
- [SALYPATH: A Deep-Based Architecture for visual attention prediction](https://mlsanity.com/paper/2107.00559/) (2021)
- [The Lipreading Gap: Do VSR Models Perceive Visual Speech Like Human Lipreaders?](https://mlsanity.com/paper/2606.07435/) (2026)
- [An Inter-observer consistent deep adversarial training for visual scanpath prediction](https://mlsanity.com/paper/2211.07336/) (2022)
- [ChartGaze: Enhancing Chart Understanding in LVLMs with Eye-Tracking Guided Attention Refinement](https://mlsanity.com/paper/2509.13282/) (2025)
- [DTFSal: Audio-Visual Dynamic Token Fusion for Video Saliency Prediction](https://mlsanity.com/paper/2504.10070/) (2025)
- [Training Better Deep Learning Models Using Human Saliency](https://mlsanity.com/paper/2410.16190/) (2024)

## Abstract (arXiv)

Recent advances in deep learning have pushed the performances of visual saliency models way further than it has ever been. Numerous models in the literature present new ways to design neural networks, to arrange gaze pattern data, or to extract as much high and low-level image features as possible in order to create the best saliency representation. However, one key part of a typical deep learning model is often neglected: the choice of the loss function. In this work, we explore some of the most popular loss functions that are used in deep saliency models. We demonstrate that on a fixed network architecture, modifying the loss function can significantly improve (or depreciate) the results, hence emphasizing the importance of the choice of the loss function when designing a model. We also introduce new loss functions that have never been used for saliency prediction to our knowledge. And finally, we show that a linear combination of several well-chosen loss functions leads to significant improvements in performances on different datasets as well as on a different network architecture, hence demonstrating the robustness of a combined metric.