# Object-aware Adaptive-Positivity Learning for Audio-Visual Question Answering

arXiv: https://arxiv.org/abs/2312.12816 · HTML: https://mlsanity.com/paper/2312.12816/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 28 references that are in the map.
- First posted on arXiv 20 Dec 2023.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [TVQA: Localized, Compositional Video Question Answering](https://mlsanity.com/paper/1809.01696/) (2018)
- [Learning to Answer Questions in Dynamic Audio-Visual Scenarios](https://mlsanity.com/paper/2203.14072/) (2022)
- [Pano-AVQA: Grounded Audio-Visual Question Answering on 360^circ Videos](https://mlsanity.com/paper/2110.05122/) (2021)
- [Positive Sample Propagation along the Audio-Visual Event Line](https://mlsanity.com/paper/2104.00239/) (2021)
- [Video as Conditional Graph Hierarchy for Multi-Granular Question Answering](https://mlsanity.com/paper/2112.06197/) (2021)
- [A Simple Baseline for Audio-Visual Scene-Aware Dialog](https://mlsanity.com/paper/1904.05876/) (2019)
- [Contrastive Positive Sample Propagation along the Audio-Visual Event Line](https://mlsanity.com/paper/2211.09980/) (2022)
- [Multimodal Variational Auto-encoder based Audio-Visual Segmentation](https://mlsanity.com/paper/2310.08303/) (2023)
- [Hierarchical Object-oriented Spatio-Temporal Reasoning for Video Question Answering](https://mlsanity.com/paper/2106.13432/) (2021)
- [ViGT: Proposal-free Video Grounding with Learnable Token in Transformer](https://mlsanity.com/paper/2308.06009/) (2023)
- [Improving Audio-Visual Video Parsing with Pseudo Visual Labels](https://mlsanity.com/paper/2303.02344/) (2023)
- [Multi-Scale Attention for Audio Question Answering](https://mlsanity.com/paper/2305.17993/) (2023)

## Built on by (most cited first, 19 of 19)

- [Towards Open-Vocabulary Audio-Visual Event Localization](https://mlsanity.com/paper/2411.11278/) (2024)
- [Label-anticipated Event Disentanglement for Audio-Visual Video Parsing](https://mlsanity.com/paper/2407.08126/) (2024)
- [Advancing Weakly-Supervised Audio-Visual Video Parsing via Segment-wise Pseudo Labeling](https://mlsanity.com/paper/2406.00919/) (2024)
- [Boosting Audio Visual Question Answering via Key Semantic-Aware Cues](https://mlsanity.com/paper/2407.20693/) (2024)
- [Question-Aware Gaussian Experts for Audio-Visual Question Answering](https://mlsanity.com/paper/2503.04459/) (2025)
- [Mettle: Meta-Token Learning for Memory-Efficient Audio-Visual Adaptation](https://mlsanity.com/paper/2506.23271/) (2025)
- [Dynamic Derivation and Elimination: Audio Visual Segmentation with Enhanced Audio Semantics](https://mlsanity.com/paper/2503.12840/) (2025)
- [SimToken: A Simple Baseline for Referring Audio-Visual Segmentation](https://mlsanity.com/paper/2509.17537/) (2025)
- [PreFM: Online Audio-Visual Event Parsing via Predictive Future Modeling](https://mlsanity.com/paper/2505.23155/) (2025)
- [AV-Master: Dual-Path Comprehensive Perception Makes Better Audio-Visual Question Answering](https://mlsanity.com/paper/2510.18346/) (2025)
- [Query-Guided Spatial-Temporal-Frequency Interaction for Music Audio-Visual Question Answering](https://mlsanity.com/paper/2601.19821/) (2026)
- [Audit After Segmentation: Reference-Free Mask Quality Assessment for Language-Referred Audio-Visual Segmentation](https://mlsanity.com/paper/2602.03892/) (2026)
- [Knowing When to Answer: Adaptive Confidence Refinement for Reliable Audio-Visual Question Answering](https://mlsanity.com/paper/2602.04924/) (2026)
- [RVAFM: Re-parameterizing Vertical Attention Fusion Module for Handwritten Paragraph Text Recognition](https://mlsanity.com/paper/2503.03104/) (2025)
- [Face-Guided Sentiment Boundary Enhancement for Weakly-Supervised Temporal Sentiment Localization](https://mlsanity.com/paper/2603.14750/) (2026)
- [Towards Multilingual Audio-Visual Question Answering](https://mlsanity.com/paper/2406.09156/) (2024)
- [Multi-Modal Scene Graph with Kolmogorov-Arnold Experts for Audio-Visual Question Answering](https://mlsanity.com/paper/2511.23304/) (2025)
- [Q-TriM: Question-Guided Tri-Modal Attention for Audio-Visual Question Answering](https://mlsanity.com/paper/2607.03825/) (2026)
- [Motion-Aware Reasoning from Speech to Mask Tracks: Runner-up Solution for the MeViS-Audio Track of the 8th LSVOS Challenge 2026](https://mlsanity.com/paper/2608.22337/) (2026)

## Abstract (arXiv)

This paper focuses on the Audio-Visual Question Answering (AVQA) task that aims to answer questions derived from untrimmed audible videos. To generate accurate answers, an AVQA model is expected to find the most informative audio-visual clues relevant to the given questions. In this paper, we propose to explicitly consider fine-grained visual objects in video frames (object-level clues) and explore the multi-modal relations(i.e., the object, audio, and question) in terms of feature interaction and model optimization. For the former, we present an end-to-end object-oriented network that adopts a question-conditioned clue discovery module to concentrate audio/visual modalities on respective keywords of the question and designs a modality-conditioned clue collection module to highlight closely associated audio segments or visual objects. For model optimization, we propose an object-aware adaptive-positivity learning strategy that selects the highly semantic-matched multi-modal pair as positivity. Specifically, we design two object-aware contrastive loss functions to identify the highly relevant question-object pairs and audio-object pairs, respectively. These selected pairs are constrained to have larger similarity values than the mismatched pairs. The positivity-selecting process is adaptive as the positivity pairs selected in each video frame may be different. These two object-aware objectives help the model understand which objects are exactly relevant to the question and which are making sounds. Extensive experiments on the MUSIC-AVQA dataset demonstrate the proposed method is effective in finding favorable audio-visual clues and also achieves new state-of-the-art question-answering performance.