# Are you really listening? Boosting Perceptual Awareness in Music-QA Benchmarks

arXiv: https://arxiv.org/abs/2504.00369 · HTML: https://mlsanity.com/paper/2504.00369/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Apr 2025.

## Built on by (most cited first, 18 of 18)

- [Qwen3-Omni Technical Report](https://mlsanity.com/paper/2509.17765/) (2025)
- [Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models](https://mlsanity.com/paper/2507.08128/) (2025)
- [Qwen3.5-Omni Technical Report](https://mlsanity.com/paper/2604.15804/) (2026)
- [Towards Holistic Evaluation of Large Audio-Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2505.15957/) (2025)
- [Music Flamingo: Scaling Music Understanding in Audio Language Models](https://mlsanity.com/paper/2511.10289/) (2025)
- [Audio Flamingo Next: Next-Generation Open Audio-Language Models for Speech, Sound, and Music](https://mlsanity.com/paper/2604.10905/) (2026)
- [WildScore: Benchmarking MLLMs in-the-Wild Symbolic Music Reasoning](https://mlsanity.com/paper/2509.04744/) (2025)
- [Investigating Modality Contribution in Audio LLMs for Music](https://mlsanity.com/paper/2509.20641/) (2025)
- [Musical Score Understanding Benchmark: Evaluating Large Language Models' Comprehension of Complete Musical Scores](https://mlsanity.com/paper/2511.20697/) (2025)
- [Qwen3.8-Omni: Towards Native Omni-Modal Agents](https://mlsanity.com/paper/2609.25611/) (2026)
- [HumMusQA: A Human-written Music Understanding QA Benchmark Dataset](https://mlsanity.com/paper/2603.27877/) (2026)
- [Evaluating Multimodal Large Language Models on Core Music Perception Tasks](https://mlsanity.com/paper/2510.22455/) (2025)
- [Assessing Factual Music Comprehension in Large Audio Language Models](https://mlsanity.com/paper/2511.05550/) (2025)
- [LabelBuddy: An Open Source Music and Audio Language Annotation Tagging Tool Using AI Assistance](https://mlsanity.com/paper/2603.04293/) (2026)
- [Beyond Binary Instrument QA: Probing Instrument Grounding in Music Audio-Language Models](https://mlsanity.com/paper/2606.31338/) (2026)
- [Auditory Illusion Benchmark for Large Audio Language Models](https://mlsanity.com/paper/2609.02277/) (2026)
- [Don't CLAP: Are Music-Text Models Bag-of-Words?](https://mlsanity.com/paper/2609.30540/) (2026)
- [What Do Audio-Language Judges of Generated Music Listen For? Four Probes with Controls from Expert Ratings](https://mlsanity.com/paper/2610.13247/) (2026)

## Abstract (arXiv)

Large Audio Language Models (LALMs), where pretrained text LLMs are finetuned with audio input, have made remarkable progress in music understanding. However, current evaluation methodologies exhibit critical limitations: on the leading Music Question Answering benchmark, MuchoMusic, text-only LLMs without audio perception capabilities achieve surprisingly high accuracy of up to 56.4%, on par or above most LALMs. Furthermore, when presented with random Gaussian noise instead of actual audio, LALMs still perform significantly above chance. These findings suggest existing benchmarks predominantly assess reasoning abilities rather than audio perception. To overcome this challenge, we present RUListening: Robust Understanding through Listening, a framework that enhances perceptual evaluation in Music-QA benchmarks. We introduce the Perceptual Index (PI), a quantitative metric that measures a question's reliance on audio perception by analyzing log probability distributions from text-only language models. Using this metric, we generate synthetic, challenging distractors to create QA pairs that necessitate genuine audio perception. When applied to MuchoMusic, our filtered dataset successfully forces models to rely on perceptual information-text-only LLMs perform at chance levels, while LALMs similarly deteriorate when audio inputs are replaced with noise. These results validate our framework's effectiveness in creating benchmarks that more accurately evaluate audio perception capabilities.