mlsanityData to 14 Oct 2026

HumMusQA: A Human-written Music Understanding QA Benchmark Dataset

arXiv 2603.27877 ↗First posted 29 Mar 2026cs.CL cs.SDcited by 1

Not on the map. Why?

Abstract (arXiv)

The evaluation of music understanding in Large Audio-Language Models (LALMs) requires a rigorously defined benchmark that truly tests whether models can perceive and interpret music, a standard that current data methodologies frequently fail to meet. This paper introduces a meticulously structured approach to music evaluation, proposing a new dataset of 320 hand-written questions curated and validated by experts with musical training, arguing that such focused, manual curation is superior for probing complex audio comprehension. To demonstrate the use of the dataset, we benchmark six state-of-the-art LALMs and additionally test their robustness to uni-modal shortcuts.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 19 of its 30 references that are in the map. First posted on arXiv 29 Mar 2026.

Builds on

19 of 30 references.

References · 19

  1. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 832
  2. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
  3. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 486
  4. 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmarkcited by 288
  5. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 263
  6. 2024AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehensioncited by 220
  7. 2025Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Modelscited by 219
  8. 2025Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitiescited by 133
  9. 2023Music Understanding LLaMA: Advancing Text-to-Music Generation with Question Answering and Captioningcited by 77
  10. 2023The Song Describer Dataset: a Corpus of Audio Captions for Music-and-Language Evaluationcited by 68
  11. 2024ChatMusician: Understanding and Generating Music Intrinsically with LLMcited by 68
  12. 2023MusiLingo: Bridging Music and Text with Pre-trained Language Models for Music Captioning and Query Responsecited by 45
7 more
  1. 2024MuChoMusic: Evaluating Music Understanding in Multimodal Audio-Language Modelscited by 44
  2. 2025Music Flamingo: Scaling Music Understanding in Audio Language Modelscited by 21
  3. 2025Are you really listening? Boosting Perceptual Awareness in Music-QA Benchmarkscited by 16
  4. 2025AudSemThinker: Enhancing Audio-Language Models through Reasoning over Semantics of Soundcited by 11
  5. 2025MusGO: A Community-Driven Framework For Assessing Openness in Music-Generative AIcited by 7
  6. 2025OMAR-RQ: Open Music Audio Representation Model Trained with Multi-Feature Masked Token Predictioncited by 5
  7. 2025Hearing the Order: Investigating Position Bias in Large Audio-Language Modelscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Qwen3.8-Omni: Towards Native Omni-Modal Agentscited by 2