# DAVE: A Deep Audio-Visual Embedding for Dynamic Saliency Prediction

arXiv: https://arxiv.org/abs/1905.10693 · HTML: https://mlsanity.com/paper/1905.10693/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 May 2019.

## Built on by (most cited first, 21 of 21)

- [Learning in Audio-visual Context: A Review, Analysis, and New Perspective](https://mlsanity.com/paper/2208.09579/) (2022)
- [Listen to Look into the Future: Audio-Visual Egocentric Gaze Anticipation](https://mlsanity.com/paper/2305.03907/) (2023)
- [CASP-Net: Rethinking Video Saliency Prediction from an Audio-VisualConsistency Perceptual Perspective](https://mlsanity.com/paper/2303.06357/) (2023)
- [DiffSal: Joint Audio and Video Learning for Diffusion Saliency Prediction](https://mlsanity.com/paper/2403.01226/) (2024)
- [CAD -- Contextual Multi-modal Alignment for Dynamic AVQA](https://mlsanity.com/paper/2310.16754/) (2023)
- [NTIRE 2026 Challenge on Video Saliency Prediction: Methods and Results](https://mlsanity.com/paper/2604.14816/) (2026)
- [A Comprehensive Survey on Video Saliency Detection with Auditory Information: the Audio-visual Consistency Perceptual is the Key!](https://mlsanity.com/paper/2206.13390/) (2022)
- [Audio-visual Saliency for Omnidirectional Videos](https://mlsanity.com/paper/2311.05190/) (2023)
- [AIM 2024 Challenge on Video Saliency Prediction: Methods and Results](https://mlsanity.com/paper/2409.14827/) (2024)
- [Learning to Predict Salient Faces: A Novel Visual-Audio Saliency Model](https://mlsanity.com/paper/2103.15438/) (2021)
- [How Does Audio Influence Visual Attention in Omnidirectional Videos? Database and Model](https://mlsanity.com/paper/2408.05411/) (2024)
- [ASOD60K: An Audio-Induced Salient Object Detection Dataset for Panoramic Videos](https://mlsanity.com/paper/2107.11629/) (2021)
- [Where to look at the movies : Analyzing visual attention to understand movie editing](https://mlsanity.com/paper/2102.13378/) (2021)
- [Audio-Visual Collaborative Representation Learning for Dynamic Saliency Prediction](https://mlsanity.com/paper/2109.08371/) (2021)
- [Joint Learning of Visual-Audio Saliency Prediction and Sound Source Localization on Multi-face Videos](https://mlsanity.com/paper/2111.08567/) (2021)
- [Weakly Supervised Visual-Auditory Fixation Prediction with Multigranularity Perception](https://mlsanity.com/paper/2112.13697/) (2021)
- [NPF-200: A Multi-Modal Eye Fixation Dataset and Method for Non-Photorealistic Videos](https://mlsanity.com/paper/2308.12163/) (2023)
- [Relevance-guided Audio Visual Fusion for Video Saliency Prediction](https://mlsanity.com/paper/2411.11454/) (2024)
- [DTFSal: Audio-Visual Dynamic Token Fusion for Video Saliency Prediction](https://mlsanity.com/paper/2504.10070/) (2025)
- [Dual Domain-Adversarial Learning for Audio-Visual Saliency Prediction](https://mlsanity.com/paper/2208.05220/) (2022)
- [ViDaS Video Depth-aware Saliency Network](https://mlsanity.com/paper/2305.11729/) (2023)

## Abstract (arXiv)

This paper studies audio-visual deep saliency prediction. It introduces a conceptually simple and effective Deep Audio-Visual Embedding for dynamic saliency prediction dubbed ``DAVE" in conjunction with our efforts towards building an Audio-Visual Eye-tracking corpus named ``AVE". Despite existing a strong relation between auditory and visual cues for guiding gaze during perception, video saliency models only consider visual cues and neglect the auditory information that is ubiquitous in dynamic scenes. Here, we investigate the applicability of audio cues in conjunction with visual ones in predicting saliency maps using deep neural networks. To this end, the proposed model is intentionally designed to be simple. Two baseline models are developed on the same architecture which consists of an encoder-decoder. The encoder projects the input into a feature space followed by a decoder that infers saliency. We conduct an extensive analysis on different modalities and various aspects of multi-model dynamic saliency prediction. Our results suggest that (1) audio is a strong contributing cue for saliency prediction, (2) salient visible sound-source is the natural cause of the superiority of our Audio-Visual model, (3) richer feature representations for the input space leads to more powerful predictions even in absence of more sophisticated saliency decoders, and (4) Audio-Visual model improves over 53.54% of the frames predicted by the best Visual model (our baseline). Our endeavour demonstrates that audio is an important cue that boosts dynamic video saliency prediction and helps models to approach human performance. The code is available at https://github.com/hrtavakoli/DAVE