mlsanityData to 7 Oct 2026

Learning to Answer Questions in Dynamic Audio-Visual Scenarios

arXiv 2203.14072 ↗First posted 26 Mar 2022cs.CVcited by 188

Not on the map. Why?

Cited by 40 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this paper, we focus on the Audio-Visual Question Answering (AVQA) task, which aims to answer questions regarding different visual objects, sounds, and their associations in videos. The problem requires comprehensive multimodal understanding and spatio-temporal reasoning over audio-visual scenes. To benchmark this task and facilitate our study, we introduce a large-scale MUSIC-AVQA dataset, which contains more than 45K question-answer pairs covering 33 different question templates spanning over different modalities and question types. We develop several baselines and introduce a spatio-temporal grounded audio-visual network for the AVQA problem. Our results demonstrate that AVQA benefits from multisensory perception and our model outperforms recent A-, V-, and AVQA approaches. We believe that our built dataset has the potential to serve as testbed for evaluating and promoting progress in audio-visual scene understanding and spatio-temporal reasoning. Code and dataset: http://gewu-lab.github.io/MUSIC-AVQA/

Key facts. Cited by 188 papers in the map (data through 7 Oct 2026). Builds on 49 of its 67 references that are in the map. First posted on arXiv 26 Mar 2022.

Builds on

49 of 67 references.

References · 49

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  3. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  4. 2015VQA: Visual Question Answeringcited by 2,489
  5. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  6. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  7. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  8. 2018TVQA: Localized, Compositional Video Question Answeringcited by 463
  9. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  10. 2015Neural Module Networkscited by 375
  11. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 338
  12. 2015Towards AI-Complete Question Answering: A Set of Prerequisite Toy Taskscited by 309
37 more
  1. 2015MovieQA: Understanding Stories in Movies through Question-Answeringcited by 304
  2. 2016Hierarchical Question-Image Co-Attention for Visual Question Answeringcited by 278
  3. 2018The Sound of Pixelscited by 258
  4. 2018Learning to Localize Sound Source in Visual Scenescited by 188
  5. 2021MultiModalQA: Complex Question Answering over Text, Tables and Imagescited by 177
  6. 2021Localizing Visual Sounds the Hard Waycited by 169
  7. 2019Listen to Look: Action Recognition by Previewing Audiocited by 163
  8. 2020Hierarchical Conditional Relation Networks for Video Question Answeringcited by 163
  9. 2021Self-supervised Video Object Segmentation by Motion Groupingcited by 147
  10. 2019Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answeringcited by 146
  11. 2020Unified Multisensory Perception: Weakly-Supervised Audio-Visual Video Parsingcited by 146
  12. 2019Audio-Visual Scene-Aware Dialogcited by 122
  13. 2020Multiple Sound Sources Localization from Coarse to Finecited by 118
  14. 2018Learning to Separate Object Sounds by Watching Unlabeled Videocited by 114
  15. 2021VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistencycited by 114
  16. 2018Dynamic Fusion with Intra- and Inter- Modality Attention Flow for Visual Question Answeringcited by 113
  17. 2020Multi-modal Dense Video Captioningcited by 113
  18. 2020Music Gesture for Visual Sound Separationcited by 112
  19. 2015Yin and Yang: Balancing and Answering Binary Visual Questionscited by 109
  20. 2020Discriminative Sounding Objects Localization via Self-supervised Audiovisual Matchingcited by 104
  21. 2021Pano-AVQA: Grounded Audio-Visual Question Answering on 360^circ Videoscited by 86
  22. 2021Positive Sample Propagation along the Audio-Visual Event Linecited by 69
  23. 2021Cyclic Co-Learning of Sounding Object Visual Grounding and Sound Separationcited by 59
  24. 2020Sep-Stereo: Visually Guided Stereophonic Audio Generation by Associating Source Separationcited by 55
  25. 2019Recursive Visual Sound Separation Using Minus-Plus Netcited by 53
  26. 2019A Simple Baseline for Audio-Visual Scene-Aware Dialogcited by 52
  27. 2018Learning Conditioned Graph Structures for Interpretable Visual Question Answeringcited by 45
  28. 2016Image Captioning and Visual Question Answering Based on Attributes and External Knowledgecited by 44
  29. 2019Vision-Infused Deep Audio Inpaintingcited by 40
  30. 2019Temporal Reasoning via Audio Question Answeringcited by 38
  31. 2020Modality Shifting Attention Network for Multi-modal Video Question Answeringcited by 38
  32. 2021Class-aware Sounding Objects Localization via Audiovisual Correspondencecited by 32
  33. 2022Visual Sound Localization in the Wild by Cross-Modal Interference Erasingcited by 19
  34. 2020Knowledge-Based Video Question Answering with Unsupervised Scene Descriptionscited by 15
  35. 2020Describing Unseen Videos via Multi-Modal Cooperative Dialog Agentscited by 11
  36. 2017Speech-Based Visual Question Answeringcited by 10
  37. 2021Multi-level Attention Fusion Network for Audio-visual Event Recognitioncited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 27; 2024: 44; 2025: 62; 2026: 51.

188 papers.

Sort:

By area:

  1. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  2. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  3. 2024AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehensioncited by 215
  4. 2025Kimi-Audio Technical Reportcited by 210
  5. 2023OneLLM: One Framework to Align All Modalities with Languagecited by 185
  6. 2024Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilitiescited by 183
  7. 2023VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Datasetcited by 168
  8. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 138
  9. 2025Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitiescited by 127
  10. 2025Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalitiescited by 101
  11. 2024OmniBench: Towards The Future of Universal Omni-Language Modelscited by 100
  12. 2023X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoningcited by 78
38 more
  1. 2023Audio-Visual LLM for Video Understandingcited by 76
  2. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 73
  3. 2022Vision Transformers are Parameter-Efficient Audio-Visual Learnerscited by 72
  4. 2024TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal Foundation Modelscited by 71
  5. 2023ChatBridge: Bridging Modalities with Large Language Model as a Language Catalystcited by 64
  6. 2025OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMscited by 54
  7. 2024CAT: Enhancing Multimodal Large Language Model to Answer Questions in Dynamic Audio-Visual Scenarioscited by 53
  8. 2024AVHBench: A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language Modelscited by 50
  9. 2024LongVALE: Vision-Audio-Language-Event Benchmark Towards Time-Aware Omni-Modal Perception of Long Videoscited by 49
  10. 2024AV-Odyssey Bench: Can Your Multimodal LLMs Really Understand Audio-Visual Information?cited by 46
  11. 2022Learning in Audio-visual Context: A Review, Analysis, and New Perspectivecited by 44
  12. 2024Align Anything: Training All-Modality Models to Follow Instructions with Language Feedbackcited by 38
  13. 2024Tell What You Hear From What You See -- Video to Audio Generation Through Textcited by 34
  14. 2025Audio-centric Video Understanding Benchmark without Text Shortcutcited by 33
  15. 2024Ref-AVS: Refer and Segment Objects in Audio-Visual Scenescited by 32
  16. 2024On-the-fly Modulation for Balanced Multimodal Learningcited by 32
  17. 2023Cross-modal Prompts: Adapting Large Pre-trained Models for Audio-Visual Downstream Taskscited by 31
  18. 2023Progressive Spatio-temporal Perception for Audio-Visual Question Answeringcited by 29
  19. 2023Tackling Data Bias in MUSIC-AVQA: Crafting a Balanced Dataset for Unbiased Question-Answeringcited by 28
  20. 2025Temporal Working Memory: Query-Guided Segment Refinement for Enhanced Multimodal Understandingcited by 28
  21. 2024Meerkat: Audio-Visual Large Language Model for Grounding in Space and Timecited by 26
  22. 2024Towards Open-Vocabulary Audio-Visual Event Localizationcited by 26
  23. 2025AVCD: Mitigating Hallucinations in Audio-Visual Large Language Models through Contrastive Decodingcited by 26
  24. 2024Label-anticipated Event Disentanglement for Audio-Visual Video Parsingcited by 25
  25. 2025Human Motion Video Generation: A Surveycited by 25
  26. 2022Vision+X: A Survey on Multimodal Learning in the Light of Datacited by 22
  27. 2023Class-Incremental Grouping Network for Continual Audio-Visual Learningcited by 22
  28. 2024From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalitiescited by 22
  29. 2026LVOmniBench: Pioneering Long Audio-Video Understanding Evaluation for Omnimodal LLMscited by 22
  30. 2023Audio-Visual Instance Segmentationcited by 21
  31. 2025Uni-MoE-2.0-Omni: Scaling Language-Centric Omnimodal Large Model with Advanced MoE, Training and Datacited by 21
  32. 2024Advancing Weakly-Supervised Audio-Visual Video Parsing via Segment-wise Pseudo Labelingcited by 20
  33. 2025Robust Disentangled Counterfactual Learning for Physical Audiovisual Commonsense Reasoningcited by 20
  34. 2023Object-aware Adaptive-Positivity Learning for Audio-Visual Question Answeringcited by 19
  35. 2024Play to the Score: Stage-Guided Dynamic Multi-Sensory Fusion for Robotic Manipulationcited by 19
  36. 2025AVTrustBench: Assessing and Enhancing Reliability and Robustness in Audio-Visual LLMscited by 19
  37. 2025video-SALMONN-o1: Reasoning-enhanced Audio-visual Large Language Modelcited by 19
  38. 2025See, Hear, and Understand: Benchmarking Audiovisual Human Speech Understanding in Multimodal Large Language Modelscited by 19

50 most cited of 188 All 188 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025FortisAVQA and MAVEN: a Benchmark Dataset and Debiasing Framework for Robust Multimodal Reasoning
  2. 2026Listen, See and Track: Spatio-Temporal Audio-Visual Sound Event Reasoning for Omni-Modal Language Models
  3. 2026OmniVideo-100K: A Dataset for Audio-Visual Reasoning through Structured Scripts and Evidence Chains
  4. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge
  5. 2024VQA2: Visual Question Answering for Video Quality Assessment
  6. 2025R-AVST: Empowering Video-LLMs with Fine-Grained Spatio-Temporal Reasoning in Complex Audio-Visual Scenarios
  7. 2025JointAVBench: A Benchmark for Joint Audio-Visual Reasoning Evaluation
  8. 2023Language Guided Visual Question Answering: Elevate Your Multimodal Language Model Using Knowledge-Enriched Prompts