# Modeling Saliency Dataset Bias

arXiv: https://arxiv.org/abs/2505.10169 · HTML: https://mlsanity.com/paper/2505.10169/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 76 references that are in the map.
- First posted on arXiv 15 May 2025.

## Builds on (most cited first, 23 of 23)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [When and why vision-language models behave like bags-of-words, and what to do about it?](https://mlsanity.com/paper/2210.01936/) (2022)
- [Improving robustness against common corruptions by covariate shift adaptation](https://mlsanity.com/paper/2006.16971/) (2020)
- [What's "up" with vision-language models? Investigating their struggle with spatial reasoning](https://mlsanity.com/paper/2310.19785/) (2023)
- [Unified Image and Video Saliency Modeling](https://mlsanity.com/paper/2003.05477/) (2020)
- [Deep Gaze I: Boosting Saliency Prediction with Feature Maps Trained on ImageNet](https://mlsanity.com/paper/1411.1045/) (2014)
- [DeepGaze IIE: Calibrated prediction in and out-of-domain for state-of-the-art saliency modeling](https://mlsanity.com/paper/2105.12441/) (2021)
- [CAT2000: A Large Scale Fixation Dataset for Boosting Saliency Research](https://mlsanity.com/paper/1505.03581/) (2015)
- [EML-NET:An Expandable Multi-Layer NETwork for Saliency Prediction](https://mlsanity.com/paper/1805.01047/) (2018)
- [DeepFix: A Fully Convolutional Neural Network for predicting Human Eye Fixations](https://mlsanity.com/paper/1510.02927/) (2015)
- [A Decade's Battle on Dataset Bias: Are We There Yet?](https://mlsanity.com/paper/2403.08632/) (2024)
- [Contextual Encoder-Decoder Network for Visual Saliency Prediction](https://mlsanity.com/paper/1902.06634/) (2019)
- [TranSalNet: Towards perceptually relevant visual saliency prediction](https://mlsanity.com/paper/2110.03593/) (2021)
- [Simple vs complex temporal recurrences for video saliency prediction](https://mlsanity.com/paper/1907.01869/) (2019)
- [TempSAL -- Uncovering Temporal Information for Deep Saliency Prediction](https://mlsanity.com/paper/2301.02315/) (2023)
- [ONEBench to Test Them All: Sample-Level Benchmarking Over Open-Ended Capabilities](https://mlsanity.com/paper/2412.06745/) (2024)
- [Spatial statistics and attentional dynamics in scene viewing](https://mlsanity.com/paper/1405.3270/) (2014)
- [UniAR: A Unified model for predicting human Attention and Responses on visual content](https://mlsanity.com/paper/2312.10175/) (2023)
- [Data Augmentation via Latent Diffusion for Saliency Prediction](https://mlsanity.com/paper/2409.07307/) (2024)
- [What Do Deep Saliency Models Learn about Visual Attention?](https://mlsanity.com/paper/2310.09679/) (2023)
- [A dynamical scan path model for task-dependence during scene viewing](https://mlsanity.com/paper/2112.11067/) (2021)
- [SalNAS: Efficient Saliency-prediction Neural Architecture Search with self-knowledge distillation](https://mlsanity.com/paper/2407.20062/) (2024)

## Built on by (most cited first, 4 of 4)

- [Raising the Ceiling: Better Empirical Fixation Densities for Saliency Benchmarking](https://mlsanity.com/paper/2605.03885/) (2026)
- [Attend to Anything: Foundation Model for Unified Human Attention Modeling](https://mlsanity.com/paper/2606.03540/) (2026)
- [DeepGaze3.5-VL: Modeling Scanpaths via Autoregressive Token Prediction](https://mlsanity.com/paper/2607.02083/) (2026)
- [Keep The Essentials: Efficient Reference Conditioned Generation via Token Dropping](https://mlsanity.com/paper/2606.23682/) (2026)

## Abstract (arXiv)

Recent advances in image-based saliency prediction are approaching gold standard performance levels on existing benchmarks. Despite this success, we show that predicting fixations across multiple saliency datasets remains challenging due to dataset bias. We find a significant performance drop (around 40%) when models trained on one dataset are applied to another. Surprisingly, increasing dataset diversity does not resolve this inter-dataset gap, with close to 60% attributed to dataset-specific biases. To address this remaining generalization gap, we propose a novel architecture extending a mostly dataset-agnostic encoder-decoder structure with fewer than 20 dataset-specific parameters that govern interpretable mechanisms such as multi-scale structure, center bias, and fixation spread. Adapting only these parameters to new data accounts for more than 75% of the generalization gap, with a large fraction of the improvement achieved with as few as 50 samples. Our model sets a new state-of-the-art on all three datasets of the MIT/Tuebingen Saliency Benchmark (MIT300, CAT2000, and COCO-Freeview), even when purely generalizing from unrelated datasets, but with a substantial boost when adapting to the respective training datasets. The model also provides valuable insights into spatial saliency properties, revealing complex multi-scale effects that combine both absolute and relative sizes.