# Predicting Video Saliency with Object-to-Motion CNN and Two-layer Convolutional LSTM

arXiv: https://arxiv.org/abs/1709.06316 · HTML: https://mlsanity.com/paper/1709.06316/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Sep 2017.

## Built on by (most cited first, 17 of 17)

- [Enhanced Spatio-Temporal Interaction Learning for Video Deraining: A Faster and Better Framework](https://mlsanity.com/paper/2103.12318/) (2021)
- [MEDIRL: Predicting the Visual Attention of Drivers via Maximum Entropy Deep Inverse Reinforcement Learning](https://mlsanity.com/paper/1912.07773/) (2019)
- [Temporal-Spatial Feature Pyramid for Video Saliency Detection](https://mlsanity.com/paper/2105.04213/) (2021)
- [A Comprehensive Survey on Video Saliency Detection with Auditory Information: the Audio-visual Consistency Perceptual is the Key!](https://mlsanity.com/paper/2206.13390/) (2022)
- [Learning to Predict Salient Faces: A Novel Visual-Audio Saliency Model](https://mlsanity.com/paper/2103.15438/) (2021)
- [Network Pruning via Feature Shift Minimization](https://mlsanity.com/paper/2207.02632/) (2022)
- [Towards Stable Co-saliency Detection and Object Co-segmentation](https://mlsanity.com/paper/2209.12138/) (2022)
- [A Gated Fusion Network for Dynamic Saliency Prediction](https://mlsanity.com/paper/2102.07682/) (2021)
- [Spherical Convolution empowered FoV Prediction in 360-degree Video Multicast with Limited FoV Feedback](https://mlsanity.com/paper/2201.12525/) (2022)
- [Clustered Saliency Prediction](https://mlsanity.com/paper/2207.02205/) (2022)
- [NPF-200: A Multi-Modal Eye Fixation Dataset and Method for Non-Photorealistic Videos](https://mlsanity.com/paper/2308.12163/) (2023)
- [UniST: Towards Unifying Saliency Transformer for Video Saliency Prediction and Detection](https://mlsanity.com/paper/2309.08220/) (2023)
- [Noise-Aware Video Saliency Prediction](https://mlsanity.com/paper/2104.08038/) (2021)
- [Dual Domain-Adversarial Learning for Audio-Visual Saliency Prediction](https://mlsanity.com/paper/2208.05220/) (2022)
- [ViDaS Video Depth-aware Saliency Network](https://mlsanity.com/paper/2305.11729/) (2023)
- [ViASNet: A Video Ad Saliency Network for Predicting Dynamic Saliency and Viewer Engagement](https://mlsanity.com/paper/2605.29302/) (2026)
- [Moving Forward with Video Saliency: A New Dataset and Benchmark where Motion Matters](https://mlsanity.com/paper/2610.03276/) (2026)

## Abstract (arXiv)

Over the past few years, deep neural networks (DNNs) have exhibited great success in predicting the saliency of images. However, there are few works that apply DNNs to predict the saliency of generic videos. In this paper, we propose a novel DNN-based video saliency prediction method. Specifically, we establish a large-scale eye-tracking database of videos (LEDOV), which provides sufficient data to train the DNN models for predicting video saliency. Through the statistical analysis of our LEDOV database, we find that human attention is normally attracted by objects, particularly moving objects or the moving parts of objects. Accordingly, we propose an object-to-motion convolutional neural network (OM-CNN) to learn spatio-temporal features for predicting the intra-frame saliency via exploring the information of both objectness and object motion. We further find from our database that there exists a temporal correlation of human attention with a smooth saliency transition across video frames. Therefore, we develop a two-layer convolutional long short-term memory (2C-LSTM) network in our DNN-based method, using the extracted features of OM-CNN as the input. Consequently, the inter-frame saliency maps of videos can be generated, which consider the transition of attention across video frames. Finally, the experimental results show that our method advances the state-of-the-art in video saliency prediction.