# Minimalistic Video Saliency Prediction via Efficient Decoder & Spatio Temporal Action Cues

arXiv: https://arxiv.org/abs/2502.00397 · HTML: https://mlsanity.com/paper/2502.00397/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 41 references that are in the map.
- First posted on arXiv 1 Feb 2025.

## Builds on (most cited first, 25 of 25)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices](https://mlsanity.com/paper/1707.01083/) (2017)
- [Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification](https://mlsanity.com/paper/1712.04851/) (2017)
- [AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actions](https://mlsanity.com/paper/1705.08421/) (2017)
- [What do different evaluation metrics tell us about saliency models?](https://mlsanity.com/paper/1604.03605/) (2016)
- [Actor-Context-Actor Relation Network for Spatio-Temporal Action Localization](https://mlsanity.com/paper/2006.07976/) (2020)
- [Unified Image and Video Saliency Modeling](https://mlsanity.com/paper/2003.05477/) (2020)
- [TASED-Net: Temporally-Aggregating Spatial Encoder-Decoder Network for Video Saliency Detection](https://mlsanity.com/paper/1908.05786/) (2019)
- [ViNet: Pushing the limits of Visual Modality for Audio-Visual Saliency Prediction](https://mlsanity.com/paper/2012.06170/) (2020)
- [STAViS: Spatio-Temporal AudioVisual Saliency Network](https://mlsanity.com/paper/2001.03063/) (2020)
- [Actions in the Eye: Dynamic Gaze Datasets and Learnt Saliency Models for Visual Recognition](https://mlsanity.com/paper/1312.7570/) (2013)
- [Deep Roots: Improving CNN Efficiency with Hierarchical Filter Groups](https://mlsanity.com/paper/1605.06489/) (2016)
- [Temporal-Spatial Feature Pyramid for Video Saliency Detection](https://mlsanity.com/paper/2105.04213/) (2021)
- [Spatio-Temporal Self-Attention Network for Video Saliency Prediction](https://mlsanity.com/paper/2108.10696/) (2021)
- [CASP-Net: Rethinking Video Saliency Prediction from an Audio-VisualConsistency Perceptual Perspective](https://mlsanity.com/paper/2303.06357/) (2023)
- [DiffSal: Joint Audio and Video Learning for Diffusion Saliency Prediction](https://mlsanity.com/paper/2403.01226/) (2024)
- [GAZED- Gaze-guided Cinematic Editing of Wide-Angle Monocular Video Recordings](https://mlsanity.com/paper/2010.11886/) (2020)
- [TinyHD: Efficient Video Saliency Prediction with Heterogeneous Decoders using Hierarchical Maps Distillation](https://mlsanity.com/paper/2301.04619/) (2023)
- [Transformer-based Video Saliency Prediction with High Temporal Dimension Decoding](https://mlsanity.com/paper/2401.07942/) (2024)
- [Facial Expression Recognition using Visual Saliency and Deep Learning](https://mlsanity.com/paper/1708.08016/) (2017)
- [Learning to Predict Salient Faces: A Novel Visual-Audio Saliency Model](https://mlsanity.com/paper/2103.15438/) (2021)
- [SalGaze: Personalizing Gaze Estimation Using Visual Saliency](https://mlsanity.com/paper/1910.10603/) (2019)
- [A Gated Fusion Network for Dynamic Saliency Prediction](https://mlsanity.com/paper/2102.07682/) (2021)

## Built on by (most cited first, 4 of 4)

- [DTFSal: Audio-Visual Dynamic Token Fusion for Video Saliency Prediction](https://mlsanity.com/paper/2504.10070/) (2025)
- [DriverGaze360: OmniDirectional Driver Attention with Object-Level Guidance](https://mlsanity.com/paper/2512.14266/) (2025)
- [Simplifying Knowledge Transfer in Pretrained Models](https://mlsanity.com/paper/2510.22208/) (2025)
- [ViASNet: A Video Ad Saliency Network for Predicting Dynamic Saliency and Viewer Engagement](https://mlsanity.com/paper/2605.29302/) (2026)

## Abstract (arXiv)

This paper introduces ViNet-S, a 36MB model based on the ViNet architecture with a U-Net design, featuring a lightweight decoder that significantly reduces model size and parameters without compromising performance. Additionally, ViNet-A (148MB) incorporates spatio-temporal action localization (STAL) features, differing from traditional video saliency models that use action classification backbones. Our studies show that an ensemble of ViNet-S and ViNet-A, by averaging predicted saliency maps, achieves state-of-the-art performance on three visual-only and six audio-visual saliency datasets, outperforming transformer-based models in both parameter efficiency and real-time performance, with ViNet-S reaching over 1000fps.