mlsanityData to 14 Oct 2026

Music Audio-Visual Question Answering Requires Specialized Multimodal Designs

arXiv 2505.20638 ↗First posted 27 May 2025cs.SD cs.CV cs.MMcited by 4

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

While recent Multimodal Large Language Models exhibit impressive capabilities for general multimodal tasks, specialized domains like music necessitate tailored approaches. Music Audio-Visual Question Answering (Music AVQA) particularly underscores this, presenting unique challenges with its continuous, densely layered audio-visual content, intricate temporal dynamics, and the critical need for domain-specific knowledge. Through a systematic analysis of Music AVQA datasets and methods, this paper identifies that specialized input processing, architectures incorporating dedicated spatial-temporal designs, and music-specific modeling strategies are critical for success in this domain. Our study provides valuable insights for researchers by highlighting effective design patterns empirically linked to strong performance, proposing concrete future directions for incorporating musical priors, and aiming to establish a robust foundation for advancing multimodal musical understanding. We aim to encourage further research in this area and provide a GitHub repository of relevant works: https://github.com/WenhaoYou1/Survey4MusicAVQA.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 91 of its 140 references that are in the map. First posted on arXiv 27 May 2025.

Builds on

91 of 140 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024CAT: Enhancing Multimodal Large Language Model to Answer Questions in Dynamic Audio-Visual Scenarios7 Mar

Other references · 90

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  8. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  9. 2025Qwen2.5-VL Technical Reportcited by 5,587
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  11. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  12. 2015VQA: Visual Question Answeringcited by 2,506
38 more
  1. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  2. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
  3. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,228
  4. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  5. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
  6. 2017Dense-Captioning Events in Videoscited by 793
  7. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
  8. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 680
  9. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 648
  10. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 645
  11. 2018MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversationscited by 643
  12. 2021AST: Audio Spectrogram Transformercited by 643
  13. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
  14. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 414
  15. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 388
  16. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 339
  17. 2016Hierarchical Question-Image Co-Attention for Visual Question Answeringcited by 278
  18. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 265
  19. 2020Just Ask: Learning to Answer Questions from Millions of Narrated Videoscited by 248
  20. 2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 236
  21. 2016MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videoscited by 194
  22. 2022Learning to Answer Questions in Dynamic Audio-Visual Scenarioscited by 191
  23. 2023OneLLM: One Framework to Align All Modalities with Languagecited by 187
  24. 2024Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilitiescited by 186
  25. 2024VITA: Towards Open-Source Interactive Omni Multimodal LLMcited by 173
  26. 2023VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Datasetcited by 169
  27. 2020Hierarchical Conditional Relation Networks for Video Question Answeringcited by 163
  28. 2019Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answeringcited by 146
  29. 2018How2: A Large-scale Dataset for Multimodal Language Understandingcited by 138
  30. 2016TGIF: A New Dataset and Benchmark on Animated GIF Descriptioncited by 124
  31. 2019Audio-Visual Scene-Aware Dialogcited by 123
  32. 2020Music Gesture for Visual Sound Separationcited by 112
  33. 2021Pano-AVQA: Grounded Audio-Visual Question Answering on 360^circ Videoscited by 86
  34. 2023Music Understanding LLaMA: Advancing Text-to-Music Generation with Question Answering and Captioningcited by 77
  35. 2022Vision Transformers are Parameter-Efficient Audio-Visual Learnerscited by 72
  36. 2024ChatMusician: Understanding and Generating Music Intrinsically with LLMcited by 68
  37. 2023ChatBridge: Bridging Modalities with Large Language Model as a Language Catalystcited by 66
  38. 2016madmom: a new Python Audio and Music Signal Processing Librarycited by 64

50 most cited of 90 All 90 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025SoundMind: RL-Incentivized Logic Reasoning for Audio-Language Modelscited by 15
  2. 2025Learning Sparsity for Effective and Efficient Music Performance Question Answeringcited by 4
  3. 2025ProtoVQA: An Adaptable Prototypical Framework for Explainable Fine-Grained Visual Question Answeringcited by 0
  4. 2026Qwen-MusicAVQA-7B: A Multimodal Model for Music Audio-Visual QAcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024MuChoMusic: Evaluating Music Understanding in Multimodal Audio-Language Models
  2. 2025Music Flamingo: Scaling Music Understanding in Audio Language Models
  3. 2025MusiXQA: Advancing Visual Music Understanding in Multimodal Large Language Models
  4. 2025ArtistMus: A Globally Diverse, Artist-Centric Benchmark for Retrieval-Augmented Music Question Answering
  5. 2026EXAM2: Extending Audio Understanding in Multilingual and Multimodal Analysis
  6. 2026MuSP-Bench: Advanced Multimodal Benchmarking of Music Understanding across Score and Performance ↗
  7. 2025WildScore: Benchmarking MLLMs in-the-Wild Symbolic Music Reasoning
  8. 2026KARMA-MV: A Benchmark for Causal Question Answering on Music Videos