# MDS-ViTNet: Improving saliency prediction for Eye-Tracking with Vision Transformer

arXiv: https://arxiv.org/abs/2405.19501 · HTML: https://mlsanity.com/paper/2405.19501/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 29 references that are in the map.
- First posted on arXiv 29 May 2024.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Transformers in Time Series: A Survey](https://mlsanity.com/paper/2202.07125/) (2022)
- [Visual Transformers: Token-based Image Representation and Processing for Computer Vision](https://mlsanity.com/paper/2006.03677/) (2020)
- [DeepViT: Towards Deeper Vision Transformer](https://mlsanity.com/paper/2103.11886/) (2021)
- [What do different evaluation metrics tell us about saliency models?](https://mlsanity.com/paper/1604.03605/) (2016)
- [Predicting Human Eye Fixations via an LSTM-based Saliency Attentive Model](https://mlsanity.com/paper/1611.09571/) (2016)
- [CAT2000: A Large Scale Fixation Dataset for Boosting Saliency Research](https://mlsanity.com/paper/1505.03581/) (2015)
- [EML-NET:An Expandable Multi-Layer NETwork for Saliency Prediction](https://mlsanity.com/paper/1805.01047/) (2018)
- [TranSalNet: Towards perceptually relevant visual saliency prediction](https://mlsanity.com/paper/2110.03593/) (2021)
- [How is Gaze Influenced by Image Transformations? Dataset and Model](https://mlsanity.com/paper/1905.06803/) (2019)
- [A Deep Spatial Contextual Long-term Recurrent Convolutional Network for Saliency Detection](https://mlsanity.com/paper/1610.01708/) (2016)
- [Improving Efficiency of Diffusion Models via Multi-Stage Framework and Tailored Multi-Decoder Architectures](https://mlsanity.com/paper/2312.09181/) (2023)
- [Multi-Decoder DPRNN: High Accuracy Source Counting and Separation](https://mlsanity.com/paper/2011.12022/) (2020)

## Built on by (most cited first, 1 of 1)

- [PRE-MAP: Personalized Reinforced Eye-tracking Multimodal LLM for High-Resolution Multi-Attribute Point Prediction](https://mlsanity.com/paper/2507.19213/) (2025)

## Abstract (arXiv)

In this paper, we present a novel methodology we call MDS-ViTNet (Multi Decoder Saliency by Vision Transformer Network) for enhancing visual saliency prediction or eye-tracking. This approach holds significant potential for diverse fields, including marketing, medicine, robotics, and retail. We propose a network architecture that leverages the Vision Transformer, moving beyond the conventional ImageNet backbone. The framework adopts an encoder-decoder structure, with the encoder utilizing a Swin transformer to efficiently embed most important features. This process involves a Transfer Learning method, wherein layers from the Vision Transformer are converted by the Encoder Transformer and seamlessly integrated into a CNN Decoder. This methodology ensures minimal information loss from the original input image. The decoder employs a multi-decoding technique, utilizing dual decoders to generate two distinct attention maps. These maps are subsequently combined into a singular output via an additional CNN model. Our trained model MDS-ViTNet achieves state-of-the-art results across several benchmarks. Committed to fostering further collaboration, we intend to make our code, models, and datasets accessible to the public.