Joint Learning of Visual-Audio Saliency Prediction and Sound Source Localization on Multi-face Videos
Not on the map. Why?
Abstract (arXiv)
Visual and audio events simultaneously occur and both attract attention. However, most existing saliency prediction works ignore the influence of audio and only consider vision modality. In this paper, we propose a multitask learning method for visual-audio saliency prediction and sound source localization on multi-face video by leveraging visual, audio and face information. Specifically, we first introduce a large-scale database of multi-face video in visual-audio condition (MVVA), containing eye-tracking data and sound source annotations. Using this database, we find that sound influences human attention, and conversly attention offers a cue to determine sound source on multi-face video. Guided by these findings, a visual-audio multi-task network (VAM-Net) is introduced to predict saliency and locate sound source. VAM-Net consists of three branches corresponding to visual, audio and face modalities. Visual branch has a two-stream architecture to capture spatial and temporal information. Face and audio branches encode audio signals and faces, respectively. Finally, a spatio-temporal multi-modal graph (STMG) is constructed to model the interaction among multiple faces. With joint optimization of these branches, the intrinsic correlation of the tasks of saliency prediction and sound source localization is utilized and their performance is boosted by each other. Experiments show that the proposed method outperforms 12 state-of-the-art saliency prediction methods, and achieves competitive results in sound source localization.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 34 of its 71 references that are in the map. First posted on arXiv 5 Nov 2021.
Builds on
34 of 71 references.
References · 34
- 2017Attention Is All You Needcited by 60,361
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
- 2017Graph Attention Networkscited by 8,923
- 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
- 2015Learning Deep Features for Discriminative Localizationcited by 2,436
- 2015Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcastingcited by 1,436
- 2015FlowNet: Learning Optical Flow with Convolutional Networkscited by 1,059
- 2016Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networkscited by 903
- 2017Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?cited by 551
- 2017Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classificationcited by 521
- 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
- 2019Multi-Label Image Recognition with Graph Convolutional Networkscited by 322
22 more
- 2018The Sound of Pixelscited by 259
- 2017Objects that Soundcited by 256
- 2016SoundNet: Learning Sound Representations from Unlabeled Videocited by 247
- 2018Learning to Localize Sound Source in Visual Scenescited by 189
- 2016What do different evaluation metrics tell us about saliency models?cited by 185
- 2019Understanding Attention and Generalization in Graph Neural Networkscited by 169
- 2018Learning to Separate Object Sounds by Watching Unlabeled Videocited by 114
- 2020Discriminative Sounding Objects Localization via Self-supervised Audiovisual Matchingcited by 104
- 2016Predicting Human Eye Fixations via an LSTM-based Saliency Attentive Modelcited by 91
- 2019AVA-ActiveSpeaker: An Audio-Visual Dataset for Active Speaker Detectioncited by 83
- 2020Unified Image and Video Saliency Modelingcited by 75
- 2017Deep Visual Attention Predictioncited by 70
- 2018Revisiting Video Saliency: A Large-scale Benchmark and a New Modelcited by 67
- 2017SalGAN: Visual Saliency Prediction with Generative Adversarial Networkscited by 65
- 2019TASED-Net: Temporally-Aggregating Spatial Encoder-Decoder Network for Video Saliency Detectioncited by 59
- 2020ViNet: Pushing the limits of Visual Modality for Audio-Visual Saliency Predictioncited by 43
- 2020Active Speakers in Contextcited by 41
- 2019Learning to Localize Sound Sources in Visual Scenes: Analysis and Applicationscited by 31
- 2020Hierarchical Domain-Adapted Feature Learning for Video Saliency Predictioncited by 24
- 2016Cross-modal Supervision for Learning Active Speaker Detection in Videocited by 21
- 2019DAVE: A Deep Audio-Visual Embedding for Dynamic Saliency Predictioncited by 21
- 2021Learning to Predict Salient Faces: A Novel Visual-Audio Saliency Modelcited by 6
Built on by
Citing papers per year: 2025: 1.
1 paper.