SaSR-Net: Source-Aware Semantic Representation Network for Enhancing Audio-Visual Question Answering
Not on the map. Why?
Abstract (arXiv)
Audio-Visual Question Answering (AVQA) is a challenging task that involves answering questions based on both auditory and visual information in videos. A significant challenge is interpreting complex multi-modal scenes, which include both visual objects and sound sources, and connecting them to the given question. In this paper, we introduce the Source-aware Semantic Representation Network (SaSR-Net), a novel model designed for AVQA. SaSR-Net utilizes source-wise learnable tokens to efficiently capture and align audio-visual elements with the corresponding question. It streamlines the fusion of audio and visual information using spatial and temporal attention mechanisms to identify answers in multi-modal scenes. Extensive experiments on the Music-AVQA and AVQA-Yang datasets show that SaSR-Net outperforms state-of-the-art AVQA methods.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 29 of its 39 references that are in the map. First posted on arXiv 7 Nov 2024.
Builds on
29 of 39 references.
References · 29
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 339
- 2016Hierarchical Question-Image Co-Attention for Visual Question Answeringcited by 278
- 2018The Sound of Pixelscited by 259
- 2022Learning to Answer Questions in Dynamic Audio-Visual Scenarioscited by 191
- 2019Listen to Look: Action Recognition by Previewing Audiocited by 163
- 2020Hierarchical Conditional Relation Networks for Video Question Answeringcited by 163
- 2019Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answeringcited by 146
- 2020Unified Multisensory Perception: Weakly-Supervised Audio-Visual Video Parsingcited by 146
- 2019Audio-Visual Scene-Aware Dialogcited by 123
- 2020Multiple Sound Sources Localization from Coarse to Finecited by 118
- 2021VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistencycited by 114
17 more
- 2020Multi-modal Dense Video Captioningcited by 113
- 2021Pano-AVQA: Grounded Audio-Visual Question Answering on 360^circ Videoscited by 86
- 2022Vision Transformers are Parameter-Efficient Audio-Visual Learnerscited by 72
- 2021Positive Sample Propagation along the Audio-Visual Event Linecited by 69
- 2021Cyclic Co-Learning of Sounding Object Visual Grounding and Sound Separationcited by 59
- 2019A Simple Baseline for Audio-Visual Scene-Aware Dialogcited by 52
- 2023Audio-Visual Grouping Network for Sound Localization from Mixturescited by 52
- 2019Temporal Reasoning via Audio Question Answeringcited by 39
- 2018End-to-End Audio Visual Scene-Aware Dialog using Multimodal Attention-Based Video Featurescited by 37
- 2021Class-aware Sounding Objects Localization via Audiovisual Correspondencecited by 32
- 2022iQuery: Instruments as Queries for Audio-Visual Sound Separationcited by 29
- 2023Tackling Data Bias in MUSIC-AVQA: Crafting a Balanced Dataset for Unbiased Question-Answeringcited by 29
- 2022AVE-CLIP: AudioCLIP-based Multi-window Temporal Transformer for Audio Visual Event Localizationcited by 28
- 2022Visual Sound Localization in the Wild by Cross-Modal Interference Erasingcited by 19
- 2020Describing Unseen Videos via Multi-Modal Cooperative Dialog Agentscited by 11
- 2023Target-Aware Spatio-Temporal Reasoning via Answering Questions in Dynamics Audio-Visual Scenarioscited by 8
- 2021Multi-level Attention Fusion Network for Audio-visual Event Recognitioncited by 7
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.