# DeepGaze IIE: Calibrated prediction in and out-of-domain for state-of-the-art saliency modeling

arXiv: https://arxiv.org/abs/2105.12441 · HTML: https://mlsanity.com/paper/2105.12441/

## Key facts

- Cited by 56 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 42 references that are in the map.
- First posted on arXiv 26 May 2021.

## Builds on (most cited first, 21 of 21)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty](https://mlsanity.com/paper/1912.02781/) (2019)
- [DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition](https://mlsanity.com/paper/1310.1531/) (2013)
- [Why ReLU networks yield high-confidence predictions far away from the training data and how to mitigate the problem](https://mlsanity.com/paper/1812.05720/) (2018)
- [The Secrets of Salient Object Segmentation](https://mlsanity.com/paper/1406.2807/) (2014)
- [What do different evaluation metrics tell us about saliency models?](https://mlsanity.com/paper/1604.03605/) (2016)
- [Unified Image and Video Saliency Modeling](https://mlsanity.com/paper/2003.05477/) (2020)
- [SalGAN: Visual Saliency Prediction with Generative Adversarial Networks](https://mlsanity.com/paper/1701.01081/) (2017)
- [Deep Gaze I: Boosting Saliency Prediction with Feature Maps Trained on ImageNet](https://mlsanity.com/paper/1411.1045/) (2014)
- [CAT2000: A Large Scale Fixation Dataset for Boosting Saliency Research](https://mlsanity.com/paper/1505.03581/) (2015)
- [EML-NET:An Expandable Multi-Layer NETwork for Saliency Prediction](https://mlsanity.com/paper/1805.01047/) (2018)
- [Contextual Encoder-Decoder Network for Visual Saliency Prediction](https://mlsanity.com/paper/1902.06634/) (2019)
- [How is Gaze Influenced by Image Transformations? Dataset and Model](https://mlsanity.com/paper/1905.06803/) (2019)
- [Diverse Ensembles Improve Calibration](https://mlsanity.com/paper/2007.04206/) (2020)
- [A Dilated Inception Network for Visual Saliency Prediction](https://mlsanity.com/paper/1904.03571/) (2019)

## Built on by (most cited first, 30 of 56)

- [TranSalNet: Towards perceptually relevant visual saliency prediction](https://mlsanity.com/paper/2110.03593/) (2021)
- [TempSAL -- Uncovering Temporal Information for Deep Saliency Prediction](https://mlsanity.com/paper/2301.02315/) (2023)
- [Target-absent Human Attention](https://mlsanity.com/paper/2207.01166/) (2022)
- [Towards Deeper Understanding of Camouflaged Object Detection](https://mlsanity.com/paper/2205.11333/) (2022)
- [Learning from Observer Gaze:Zero-Shot Attention Prediction Oriented by Human-Object Interaction Recognition](https://mlsanity.com/paper/2405.09931/) (2024)
- [ViSaRL: Visual Reinforcement Learning Guided by Human Saliency](https://mlsanity.com/paper/2403.10940/) (2024)
- [SUM: Saliency Unification through Mamba for Visual Attention Modeling](https://mlsanity.com/paper/2406.17815/) (2024)
- [Cal-SFDA: Source-Free Domain-adaptive Semantic Segmentation with Differentiable Expected Calibration Error](https://mlsanity.com/paper/2308.03003/) (2023)
- [Where, What, Why: Towards Explainable Driver Attention Prediction](https://mlsanity.com/paper/2506.23088/) (2025)
- [SalFBNet: Learning Pseudo-Saliency Distribution via Feedback Convolutional Networks](https://mlsanity.com/paper/2112.03731/) (2021)
- [Explain To Me: Salience-Based Explainability for Synthetic Face Detection Models](https://mlsanity.com/paper/2303.11969/) (2023)
- [UniAR: A Unified model for predicting human Attention and Responses on visual content](https://mlsanity.com/paper/2312.10175/) (2023)
- [seq-JEPA: Autoregressive Predictive Learning of Invariant-Equivariant World Models](https://mlsanity.com/paper/2505.03176/) (2025)
- [Brand Visibility in Packaging: A Deep Learning Approach for Logo Detection, Saliency-Map Prediction, and Logo Placement Analysis](https://mlsanity.com/paper/2403.02336/) (2024)
- [OAT: Object-Level Attention Transformer for Gaze Scanpath Prediction](https://mlsanity.com/paper/2407.13335/) (2024)
- [Data Augmentation via Latent Diffusion for Saliency Prediction](https://mlsanity.com/paper/2409.07307/) (2024)
- [DiffEye: Diffusion-Based Continuous Eye-Tracking Data Generation Conditioned on Natural Images](https://mlsanity.com/paper/2509.16767/) (2025)
- [Learning User Embeddings from Human Gaze for Personalised Saliency Prediction](https://mlsanity.com/paper/2403.13653/) (2024)
- [GazeFusion: Saliency-Guided Image Generation](https://mlsanity.com/paper/2407.04191/) (2024)
- [Modeling Saliency Dataset Bias](https://mlsanity.com/paper/2505.10169/) (2025)
- [Multi-view Gaze Target Estimation](https://mlsanity.com/paper/2508.05857/) (2025)
- [Foveated Diffusion: Efficient Spatially Adaptive Image and Video Generation](https://mlsanity.com/paper/2603.23491/) (2026)
- [Predicting Visual Attention in Graphic Design Documents](https://mlsanity.com/paper/2407.02439/) (2024)
- [SOS: Segment Object System for Open-World Instance Segmentation With Object Priors](https://mlsanity.com/paper/2409.14627/) (2024)
- [Context-empowered Visual Attention Prediction in Pedestrian Scenarios](https://mlsanity.com/paper/2210.16933/) (2022)
- [Learning Saliency From Fixations](https://mlsanity.com/paper/2311.14073/) (2023)
- [FSDAM: Few-Shot Driving Attention Modeling via Vision-Language Coupling](https://mlsanity.com/paper/2511.12708/) (2025)
- [Phase-Aligned RoPE for Mixed-Resolution Diffusion Transformer](https://mlsanity.com/paper/2511.19778/) (2025)
- [Clustered Saliency Prediction](https://mlsanity.com/paper/2207.02205/) (2022)
- [Multi-Scale Estimation for Omni-Directional Saliency Maps Using Learnable Equator Bias](https://mlsanity.com/paper/2309.08139/) (2023)

## Abstract (arXiv)

Since 2014 transfer learning has become the key driver for the improvement of spatial saliency prediction; however, with stagnant progress in the last 3-5 years. We conduct a large-scale transfer learning study which tests different ImageNet backbones, always using the same read out architecture and learning protocol adopted from DeepGaze II. By replacing the VGG19 backbone of DeepGaze II with ResNet50 features we improve the performance on saliency prediction from 78% to 85%. However, as we continue to test better ImageNet models as backbones (such as EfficientNetB5) we observe no additional improvement on saliency prediction. By analyzing the backbones further, we find that generalization to other datasets differs substantially, with models being consistently overconfident in their fixation predictions. We show that by combining multiple backbones in a principled manner a good confidence calibration on unseen datasets can be achieved. This new model, "DeepGaze IIE", yields a significant leap in benchmark performance in and out-of-domain with a 15 percent point improvement over DeepGaze II to 93% on MIT1003, marking a new state of the art on the MIT/Tuebingen Saliency Benchmark in all available metrics (AUC: 88.3%, sAUC: 79.4%, CC: 82.4%).