# Audio-visual Saliency for Omnidirectional Videos

arXiv: https://arxiv.org/abs/2311.05190 · HTML: https://mlsanity.com/paper/2311.05190/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 43 references that are in the map.
- First posted on arXiv 9 Nov 2023.

## Builds on (most cited first, 15 of 15)

- [Predicting Human Eye Fixations via an LSTM-based Saliency Attentive Model](https://mlsanity.com/paper/1611.09571/) (2016)
- [Cube Padding for Weakly-Supervised Saliency Prediction in 360deg Videos](https://mlsanity.com/paper/1806.01320/) (2018)
- [SalGAN: Visual Saliency Prediction with Generative Adversarial Networks](https://mlsanity.com/paper/1701.01081/) (2017)
- [AIGCIQA2023: A Large-scale Image Quality Assessment Database for AI Generated Images: from the Perspectives of Quality, Authenticity and Correspondence](https://mlsanity.com/paper/2307.00211/) (2023)
- [TASED-Net: Temporally-Aggregating Spatial Encoder-Decoder Network for Video Saliency Detection](https://mlsanity.com/paper/1908.05786/) (2019)
- [A Deep Multi-Level Network for Saliency Prediction](https://mlsanity.com/paper/1609.01064/) (2016)
- [End-to-End Human-Gaze-Target Detection with Transformers](https://mlsanity.com/paper/2203.10433/) (2022)
- [STAViS: Spatio-Temporal AudioVisual Saliency Network](https://mlsanity.com/paper/2001.03063/) (2020)
- [Predicting Head Movement in Panoramic Video: A Deep Reinforcement Learning Approach](https://mlsanity.com/paper/1710.10755/) (2017)
- [Perceptual Quality Assessment of Omnidirectional Images](https://mlsanity.com/paper/2207.02674/) (2022)
- [Confusing Image Quality Assessment: Towards Better Augmented Reality Experience](https://mlsanity.com/paper/2204.04900/) (2022)
- [Simple vs complex temporal recurrences for video saliency prediction](https://mlsanity.com/paper/1907.01869/) (2019)
- [DAVE: A Deep Audio-Visual Embedding for Dynamic Saliency Prediction](https://mlsanity.com/paper/1905.10693/) (2019)
- [Saliency in Augmented Reality](https://mlsanity.com/paper/2204.08308/) (2022)
- [Perceptual Quality Assessment of Omnidirectional Audio-visual Signals](https://mlsanity.com/paper/2307.10813/) (2023)

## Built on by (most cited first, 7 of 7)

- [AIGCOIQA2024: Perceptual Quality Assessment of AI Generated Omnidirectional Images](https://mlsanity.com/paper/2404.01024/) (2024)
- [How is Visual Attention Influenced by Text Guidance? Database and Model](https://mlsanity.com/paper/2404.07537/) (2024)
- [How Does Audio Influence Visual Attention in Omnidirectional Videos? Database and Model](https://mlsanity.com/paper/2408.05411/) (2024)
- [MVBIND: Self-Supervised Music Recommendation For Videos Via Embedding Space Binding](https://mlsanity.com/paper/2405.09286/) (2024)
- [Quality-guided Skin Tone Enhancement for Portrait Photography](https://mlsanity.com/paper/2406.15848/) (2024)
- [Quality Assessment and Distortion-aware Saliency Prediction for AI-Generated Omnidirectional Images](https://mlsanity.com/paper/2506.21925/) (2025)
- [Immersive Multimedia Communication: State-of-the-Art on eXtended Reality Streaming](https://mlsanity.com/paper/2506.10004/) (2025)

## Abstract (arXiv)

Visual saliency prediction for omnidirectional videos (ODVs) has shown great significance and necessity for omnidirectional videos to help ODV coding, ODV transmission, ODV rendering, etc.. However, most studies only consider visual information for ODV saliency prediction while audio is rarely considered despite its significant influence on the viewing behavior of ODV. This is mainly due to the lack of large-scale audio-visual ODV datasets and corresponding analysis. Thus, in this paper, we first establish the largest audio-visual saliency dataset for omnidirectional videos (AVS-ODV), which comprises the omnidirectional videos, audios, and corresponding captured eye-tracking data for three video sound modalities including mute, mono, and ambisonics. Then we analyze the visual attention behavior of the observers under various omnidirectional audio modalities and visual scenes based on the AVS-ODV dataset. Furthermore, we compare the performance of several state-of-the-art saliency prediction models on the AVS-ODV dataset and construct a new benchmark. Our AVS-ODV datasets and the benchmark will be released to facilitate future research.