# Transformer-based Video Saliency Prediction with High Temporal Dimension Decoding

arXiv: https://arxiv.org/abs/2401.07942 · HTML: https://mlsanity.com/paper/2401.07942/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 30 references that are in the map.
- First posted on arXiv 15 Jan 2024.

## Builds on (most cited first, 19 of 19)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification](https://mlsanity.com/paper/1712.04851/) (2017)
- [SoundNet: Learning Sound Representations from Unlabeled Video](https://mlsanity.com/paper/1610.09001/) (2016)
- [UniFormer: Unifying Convolution and Self-attention for Visual Recognition](https://mlsanity.com/paper/2201.09450/) (2022)
- [What do different evaluation metrics tell us about saliency models?](https://mlsanity.com/paper/1604.03605/) (2016)
- [Unified Image and Video Saliency Modeling](https://mlsanity.com/paper/2003.05477/) (2020)
- [TASED-Net: Temporally-Aggregating Spatial Encoder-Decoder Network for Video Saliency Detection](https://mlsanity.com/paper/1908.05786/) (2019)
- [Resource Efficient 3D Convolutional Neural Networks](https://mlsanity.com/paper/1904.02422/) (2019)
- [ViNet: Pushing the limits of Visual Modality for Audio-Visual Saliency Prediction](https://mlsanity.com/paper/2012.06170/) (2020)
- [Actions in the Eye: Dynamic Gaze Datasets and Learnt Saliency Models for Visual Recognition](https://mlsanity.com/paper/1312.7570/) (2013)
- [Simple vs complex temporal recurrences for video saliency prediction](https://mlsanity.com/paper/1907.01869/) (2019)
- [Hierarchical Domain-Adapted Feature Learning for Video Saliency Prediction](https://mlsanity.com/paper/2010.01220/) (2020)
- [Temporal-Spatial Feature Pyramid for Video Saliency Detection](https://mlsanity.com/paper/2105.04213/) (2021)
- [Spatio-Temporal Self-Attention Network for Video Saliency Prediction](https://mlsanity.com/paper/2108.10696/) (2021)
- [GASP: Gated Attention For Saliency Prediction](https://mlsanity.com/paper/2206.04590/) (2022)
- [UniST: Towards Unifying Saliency Transformer for Video Saliency Prediction and Detection](https://mlsanity.com/paper/2309.08220/) (2023)

## Built on by (most cited first, 7 of 7)

- [SalFoM: Dynamic Saliency Prediction with Video Foundation Models](https://mlsanity.com/paper/2404.03097/) (2024)
- [Minimalistic Video Saliency Prediction via Efficient Decoder & Spatio Temporal Action Cues](https://mlsanity.com/paper/2502.00397/) (2025)
- [DTFSal: Audio-Visual Dynamic Token Fusion for Video Saliency Prediction](https://mlsanity.com/paper/2504.10070/) (2025)
- [Exploring deep learning for Event-Based Saliency Prediction with a Transformer-based model](https://mlsanity.com/paper/2605.23790/) (2026)
- [Simplifying Knowledge Transfer in Pretrained Models](https://mlsanity.com/paper/2510.22208/) (2025)
- [BIAS: A Biologically Inspired Algorithm for Video Saliency Detection](https://mlsanity.com/paper/2604.08858/) (2026)
- [ViASNet: A Video Ad Saliency Network for Predicting Dynamic Saliency and Viewer Engagement](https://mlsanity.com/paper/2605.29302/) (2026)

## Abstract (arXiv)

In recent years, finding an effective and efficient strategy for exploiting spatial and temporal information has been a hot research topic in video saliency prediction (VSP). With the emergence of spatio-temporal transformers, the weakness of the prior strategies, e.g., 3D convolutional networks and LSTM-based networks, for capturing long-range dependencies has been effectively compensated. While VSP has drawn benefits from spatio-temporal transformers, finding the most effective way for aggregating temporal features is still challenging. To address this concern, we propose a transformer-based video saliency prediction approach with high temporal dimension decoding network (THTD-Net). This strategy accounts for the lack of complex hierarchical interactions between features that are extracted from the transformer-based spatio-temporal encoder: in particular, it does not require multiple decoders and aims at gradually reducing temporal features' dimensions in the decoder. This decoder-based architecture yields comparable performance to multi-branch and over-complicated models on common benchmarks such as DHF1K, UCF-sports and Hollywood-2.