# Query-Guided Spatial-Temporal-Frequency Interaction for Music Audio-Visual Question Answering

arXiv: https://arxiv.org/abs/2601.19821 · HTML: https://mlsanity.com/paper/2601.19821/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 37 references that are in the map.
- First posted on arXiv 27 Jan 2026.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [Token Merging: Your ViT But Faster](https://mlsanity.com/paper/2210.09461/) (2022)
- [MaPLe: Multi-modal Prompt Learning](https://mlsanity.com/paper/2210.03117/) (2022)
- [AST: Audio Spectrogram Transformer](https://mlsanity.com/paper/2104.01778/) (2021)
- [Deep Modular Co-Attention Networks for Visual Question Answering](https://mlsanity.com/paper/1906.10770/) (2019)
- [Learning to Answer Questions in Dynamic Audio-Visual Scenarios](https://mlsanity.com/paper/2203.14072/) (2022)
- [Audio-Driven Emotional Video Portraits](https://mlsanity.com/paper/2104.07452/) (2021)
- [Hierarchical Conditional Relation Networks for Video Question Answering](https://mlsanity.com/paper/2002.10698/) (2020)
- [Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answering](https://mlsanity.com/paper/1904.04357/) (2019)
- [Pano-AVQA: Grounded Audio-Visual Question Answering on 360^circ Videos](https://mlsanity.com/paper/2110.05122/) (2021)
- [Streaming Dense Video Captioning](https://mlsanity.com/paper/2404.01297/) (2024)
- [Vision Transformers are Parameter-Efficient Audio-Visual Learners](https://mlsanity.com/paper/2212.07983/) (2022)
- [CATR: Combinatorial-Dependence Audio-Queried Transformer for Audio-Visual Video Segmentation](https://mlsanity.com/paper/2309.09709/) (2023)
- [A Simple Baseline for Audio-Visual Scene-Aware Dialog](https://mlsanity.com/paper/1904.05876/) (2019)
- [A Survey of Sound Source Localization with Deep Learning Methods](https://mlsanity.com/paper/2109.03465/) (2021)
- [Temporal Reasoning via Audio Question Answering](https://mlsanity.com/paper/1911.09655/) (2019)
- [Cross-modal Prompts: Adapting Large Pre-trained Models for Audio-Visual Downstream Tasks](https://mlsanity.com/paper/2311.05152/) (2023)
- [Progressive Spatio-temporal Perception for Audio-Visual Question Answering](https://mlsanity.com/paper/2308.05421/) (2023)
- [Object-aware Adaptive-Positivity Learning for Audio-Visual Question Answering](https://mlsanity.com/paper/2312.12816/) (2023)
- [Look, Listen, and Answer: Overcoming Biases for Audio-Visual Question Answering](https://mlsanity.com/paper/2404.12020/) (2024)
- [Boosting Audio Visual Question Answering via Key Semantic-Aware Cues](https://mlsanity.com/paper/2407.20693/) (2024)
- [Patch-level Sounding Object Tracking for Audio-Visual Question Answering](https://mlsanity.com/paper/2412.10749/) (2024)
- [Question-Aware Gaussian Experts for Audio-Visual Question Answering](https://mlsanity.com/paper/2503.04459/) (2025)
- [Interactive Image Segmentation with Cross-Modality Vision Transformers](https://mlsanity.com/paper/2307.02280/) (2023)

## Built on by (most cited first, 2 of 2)

- [Semantic-Driven Scale and Spatial Selection for Efficient Cross-Modal Alignment in Referring Remote Sensing Image Segmentation](https://mlsanity.com/paper/2606.30244/) (2026)
- [Shift-Aware Identity-Guided Latent Refinement for Referring Audio--Visual Segmentation](https://mlsanity.com/paper/2610.12783/) (2026)

## Abstract (arXiv)

Audio--Visual Question Answering (AVQA) is a challenging multimodal task that requires jointly reasoning over audio, visual, and textual information in a given video to answer natural language questions. Inspired by recent advances in Video QA, many existing AVQA approaches primarily focus on visual information processing, leveraging pre-trained models to extract object-level and motion-level representations. However, in those methods, the audio input is primarily treated as complementary to video analysis, and the textual question information contributes minimally to audio--visual understanding, as it is typically integrated only in the final stages of reasoning. To address these limitations, we propose a novel Query-guided Spatial--Temporal--Frequency (QSTar) interaction method, which effectively incorporates question-guided clues and exploits the distinctive frequency-domain characteristics of audio signals, alongside spatial and temporal perception, to enhance audio--visual understanding. Furthermore, we introduce a Query Context Reasoning (QCR) block inspired by prompting, which guides the model to focus more precisely on semantically relevant audio and visual features. Extensive experiments conducted on two AVQA benchmarks demonstrate the effectiveness of our proposed method, achieving significant performance improvements over existing Audio QA, Visual QA, Video QA, and AVQA approaches. The code is released under https://github.com/lik1996/QSTar.