# ViDaS Video Depth-aware Saliency Network

arXiv: https://arxiv.org/abs/2305.11729 · HTML: https://mlsanity.com/paper/2305.11729/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 48 references that are in the map.
- First posted on arXiv 19 May 2023.

## Builds on (most cited first, 26 of 26)

- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [MegaDepth: Learning Single-View Depth Prediction from Internet Photos](https://mlsanity.com/paper/1804.00607/) (2018)
- [Holistically-Nested Edge Detection](https://mlsanity.com/paper/1504.06375/) (2015)
- [Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?](https://mlsanity.com/paper/1711.09577/) (2017)
- [Deeper Depth Prediction with Fully Convolutional Residual Networks](https://mlsanity.com/paper/1606.00373/) (2016)
- [One-Shot Video Object Segmentation](https://mlsanity.com/paper/1611.05198/) (2016)
- [What do different evaluation metrics tell us about saliency models?](https://mlsanity.com/paper/1604.03605/) (2016)
- [Predicting Human Eye Fixations via an LSTM-based Saliency Attentive Model](https://mlsanity.com/paper/1611.09571/) (2016)
- [Unified Image and Video Saliency Modeling](https://mlsanity.com/paper/2003.05477/) (2020)
- [RGB-D Salient Object Detection: A Survey](https://mlsanity.com/paper/2008.00230/) (2020)
- [Deep Visual Attention Prediction](https://mlsanity.com/paper/1705.02544/) (2017)
- [Revisiting Video Saliency: A Large-scale Benchmark and a New Model](https://mlsanity.com/paper/1801.07424/) (2018)
- [SalGAN: Visual Saliency Prediction with Generative Adversarial Networks](https://mlsanity.com/paper/1701.01081/) (2017)
- [Deep Gaze I: Boosting Saliency Prediction with Feature Maps Trained on ImageNet](https://mlsanity.com/paper/1411.1045/) (2014)
- [TASED-Net: Temporally-Aggregating Spatial Encoder-Decoder Network for Video Saliency Detection](https://mlsanity.com/paper/1908.05786/) (2019)
- [Shallow and Deep Convolutional Networks for Saliency Prediction](https://mlsanity.com/paper/1603.00845/) (2016)
- [STAViS: Spatio-Temporal AudioVisual Saliency Network](https://mlsanity.com/paper/2001.03063/) (2020)
- [Actions in the Eye: Dynamic Gaze Datasets and Learnt Saliency Models for Visual Recognition](https://mlsanity.com/paper/1312.7570/) (2013)
- [RGB-D Salient Object Detection via 3D Convolutional Neural Networks](https://mlsanity.com/paper/2101.10241/) (2021)
- [Detail Preserving Depth Estimation from a Single Image Using Attention Guided Networks](https://mlsanity.com/paper/1809.00646/) (2018)
- [Simple vs complex temporal recurrences for video saliency prediction](https://mlsanity.com/paper/1907.01869/) (2019)
- [PDNet: Prior-model Guided Depth-enhanced Network for Salient Object Detection](https://mlsanity.com/paper/1803.08636/) (2018)
- [DAVE: A Deep Audio-Visual Embedding for Dynamic Saliency Prediction](https://mlsanity.com/paper/1905.10693/) (2019)
- [Predicting Video Saliency with Object-to-Motion CNN and Two-layer Convolutional LSTM](https://mlsanity.com/paper/1709.06316/) (2017)
- [End-to-End Saliency Mapping via Probability Distribution Prediction](https://mlsanity.com/paper/1804.01793/) (2018)
- [Learning Gaze Transitions from Depth to Improve Video Saliency Estimation](https://mlsanity.com/paper/1603.03669/) (2016)

## Abstract (arXiv)

We introduce ViDaS, a two-stream, fully convolutional Video, Depth-Aware Saliency network to address the problem of attention modeling ``in-the-wild", via saliency prediction in videos. Contrary to existing visual saliency approaches using only RGB frames as input, our network employs also depth as an additional modality. The network consists of two visual streams, one for the RGB frames, and one for the depth frames. Both streams follow an encoder-decoder approach and are fused to obtain a final saliency map. The network is trained end-to-end and is evaluated in a variety of different databases with eye-tracking data, containing a wide range of video content. Although the publicly available datasets do not contain depth, we estimate it using three different state-of-the-art methods, to enable comparisons and a deeper insight. Our method outperforms in most cases state-of-the-art models and our RGB-only variant, which indicates that depth can be beneficial to accurately estimating saliency in videos displayed on a 2D screen. Depth has been widely used to assist salient object detection problems, where it has been proven to be very beneficial. Our problem though differs significantly from salient object detection, since it is not restricted to specific salient objects, but predicts human attention in a more general aspect. These two problems not only have different objectives, but also different ground truth data and evaluation metrics. To our best knowledge, this is the first competitive deep learning video saliency estimation approach that combines both RGB and Depth features to address the general problem of saliency estimation ``in-the-wild". The code will be publicly released.