# Learning Saliency From Fixations

arXiv: https://arxiv.org/abs/2311.14073 · HTML: https://mlsanity.com/paper/2311.14073/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 67 references that are in the map.
- First posted on arXiv 23 Nov 2023.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [An Introduction to Convolutional Neural Networks](https://mlsanity.com/paper/1511.08458/) (2015)
- [Attention Augmented Convolutional Networks](https://mlsanity.com/paper/1904.09925/) (2019)
- [What do different evaluation metrics tell us about saliency models?](https://mlsanity.com/paper/1604.03605/) (2016)
- [Rethinking Transformer-based Set Prediction for Object Detection](https://mlsanity.com/paper/2011.10881/) (2020)
- [Unified Image and Video Saliency Modeling](https://mlsanity.com/paper/2003.05477/) (2020)
- [Revisiting Video Saliency: A Large-scale Benchmark and a New Model](https://mlsanity.com/paper/1801.07424/) (2018)
- [SalGAN: Visual Saliency Prediction with Generative Adversarial Networks](https://mlsanity.com/paper/1701.01081/) (2017)
- [Deep Gaze I: Boosting Saliency Prediction with Feature Maps Trained on ImageNet](https://mlsanity.com/paper/1411.1045/) (2014)
- [DeepGaze IIE: Calibrated prediction in and out-of-domain for state-of-the-art saliency modeling](https://mlsanity.com/paper/2105.12441/) (2021)
- [A Deep Multi-Level Network for Saliency Prediction](https://mlsanity.com/paper/1609.01064/) (2016)
- [CAT2000: A Large Scale Fixation Dataset for Boosting Saliency Research](https://mlsanity.com/paper/1505.03581/) (2015)
- [EML-NET:An Expandable Multi-Layer NETwork for Saliency Prediction](https://mlsanity.com/paper/1805.01047/) (2018)
- [DeepFix: A Fully Convolutional Neural Network for predicting Human Eye Fixations](https://mlsanity.com/paper/1510.02927/) (2015)
- [Contextual Encoder-Decoder Network for Visual Saliency Prediction](https://mlsanity.com/paper/1902.06634/) (2019)
- [TranSalNet: Towards perceptually relevant visual saliency prediction](https://mlsanity.com/paper/2110.03593/) (2021)
- [Shallow and Deep Convolutional Networks for Saliency Prediction](https://mlsanity.com/paper/1603.00845/) (2016)
- [PathGAN: Visual Scanpath Prediction with Generative Adversarial Networks](https://mlsanity.com/paper/1809.00567/) (2018)
- [Anchor DETR: Query Design for Transformer-Based Object Detection](https://mlsanity.com/paper/2109.07107/) (2021)
- [A Deep Spatial Contextual Long-term Recurrent Convolutional Network for Saliency Detection](https://mlsanity.com/paper/1610.01708/) (2016)
- [ATSal: An Attention Based Architecture for Saliency Prediction in 360 Videos](https://mlsanity.com/paper/2011.10600/) (2020)
- [Do Saliency Models Detect Odd-One-Out Targets? New Datasets and Evaluations](https://mlsanity.com/paper/2005.06583/) (2020)

## Built on by (most cited first, 2 of 2)

- [SUM: Saliency Unification through Mamba for Visual Attention Modeling](https://mlsanity.com/paper/2406.17815/) (2024)
- [ObjectVisA-120: Object-based Visual Attention Prediction in Interactive Street-crossing Environments](https://mlsanity.com/paper/2601.13218/) (2026)

## Abstract (arXiv)

We present a novel approach for saliency prediction in images, leveraging parallel decoding in transformers to learn saliency solely from fixation maps. Models typically rely on continuous saliency maps, to overcome the difficulty of optimizing for the discrete fixation map. We attempt to replicate the experimental setup that generates saliency datasets. Our approach treats saliency prediction as a direct set prediction problem, via a global loss that enforces unique fixations prediction through bipartite matching and a transformer encoder-decoder architecture. By utilizing a fixed set of learned fixation queries, the cross-attention reasons over the image features to directly output the fixation points, distinguishing it from other modern saliency predictors. Our approach, named Saliency TRansformer (SalTR), achieves metric scores on par with state-of-the-art approaches on the Salicon and MIT300 benchmarks.