# SaSR-Net: Source-Aware Semantic Representation Network for Enhancing Audio-Visual Question Answering

arXiv: https://arxiv.org/abs/2411.04933 · HTML: https://mlsanity.com/paper/2411.04933/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 39 references that are in the map.
- First posted on arXiv 7 Nov 2024.

## Builds on (most cited first, 29 of 29)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Deep Modular Co-Attention Networks for Visual Question Answering](https://mlsanity.com/paper/1906.10770/) (2019)
- [Hierarchical Question-Image Co-Attention for Visual Question Answering](https://mlsanity.com/paper/1606.00061/) (2016)
- [The Sound of Pixels](https://mlsanity.com/paper/1804.03160/) (2018)
- [Learning to Answer Questions in Dynamic Audio-Visual Scenarios](https://mlsanity.com/paper/2203.14072/) (2022)
- [Listen to Look: Action Recognition by Previewing Audio](https://mlsanity.com/paper/1912.04487/) (2019)
- [Hierarchical Conditional Relation Networks for Video Question Answering](https://mlsanity.com/paper/2002.10698/) (2020)
- [Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answering](https://mlsanity.com/paper/1904.04357/) (2019)
- [Unified Multisensory Perception: Weakly-Supervised Audio-Visual Video Parsing](https://mlsanity.com/paper/2007.10558/) (2020)
- [Audio-Visual Scene-Aware Dialog](https://mlsanity.com/paper/1901.09107/) (2019)
- [Multiple Sound Sources Localization from Coarse to Fine](https://mlsanity.com/paper/2007.06355/) (2020)
- [VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency](https://mlsanity.com/paper/2101.03149/) (2021)
- [Multi-modal Dense Video Captioning](https://mlsanity.com/paper/2003.07758/) (2020)
- [Pano-AVQA: Grounded Audio-Visual Question Answering on 360^circ Videos](https://mlsanity.com/paper/2110.05122/) (2021)
- [Vision Transformers are Parameter-Efficient Audio-Visual Learners](https://mlsanity.com/paper/2212.07983/) (2022)
- [Positive Sample Propagation along the Audio-Visual Event Line](https://mlsanity.com/paper/2104.00239/) (2021)
- [Cyclic Co-Learning of Sounding Object Visual Grounding and Sound Separation](https://mlsanity.com/paper/2104.02026/) (2021)
- [A Simple Baseline for Audio-Visual Scene-Aware Dialog](https://mlsanity.com/paper/1904.05876/) (2019)
- [Audio-Visual Grouping Network for Sound Localization from Mixtures](https://mlsanity.com/paper/2303.17056/) (2023)
- [Temporal Reasoning via Audio Question Answering](https://mlsanity.com/paper/1911.09655/) (2019)
- [End-to-End Audio Visual Scene-Aware Dialog using Multimodal Attention-Based Video Features](https://mlsanity.com/paper/1806.08409/) (2018)
- [Class-aware Sounding Objects Localization via Audiovisual Correspondence](https://mlsanity.com/paper/2112.11749/) (2021)
- [iQuery: Instruments as Queries for Audio-Visual Sound Separation](https://mlsanity.com/paper/2212.03814/) (2022)
- [Tackling Data Bias in MUSIC-AVQA: Crafting a Balanced Dataset for Unbiased Question-Answering](https://mlsanity.com/paper/2310.06238/) (2023)
- [AVE-CLIP: AudioCLIP-based Multi-window Temporal Transformer for Audio Visual Event Localization](https://mlsanity.com/paper/2210.05060/) (2022)
- [Visual Sound Localization in the Wild by Cross-Modal Interference Erasing](https://mlsanity.com/paper/2202.06406/) (2022)
- [Describing Unseen Videos via Multi-Modal Cooperative Dialog Agents](https://mlsanity.com/paper/2008.07935/) (2020)
- [Target-Aware Spatio-Temporal Reasoning via Answering Questions in Dynamics Audio-Visual Scenarios](https://mlsanity.com/paper/2305.12397/) (2023)
- [Multi-level Attention Fusion Network for Audio-visual Event Recognition](https://mlsanity.com/paper/2106.06736/) (2021)

## Built on by (most cited first, 2 of 2)

- [AV-Master: Dual-Path Comprehensive Perception Makes Better Audio-Visual Question Answering](https://mlsanity.com/paper/2510.18346/) (2025)
- [Crab+: A Scalable and Unified Audio-Visual Scene Understanding Model with Explicit Cooperation](https://mlsanity.com/paper/2603.04128/) (2026)

## Abstract (arXiv)

Audio-Visual Question Answering (AVQA) is a challenging task that involves answering questions based on both auditory and visual information in videos. A significant challenge is interpreting complex multi-modal scenes, which include both visual objects and sound sources, and connecting them to the given question. In this paper, we introduce the Source-aware Semantic Representation Network (SaSR-Net), a novel model designed for AVQA. SaSR-Net utilizes source-wise learnable tokens to efficiently capture and align audio-visual elements with the corresponding question. It streamlines the fusion of audio and visual information using spatial and temporal attention mechanisms to identify answers in multi-modal scenes. Extensive experiments on the Music-AVQA and AVQA-Yang datasets show that SaSR-Net outperforms state-of-the-art AVQA methods.