# The MUSE Benchmark: Probing Music Perception and Auditory Relational Reasoning in Audio LLMS

arXiv: https://arxiv.org/abs/2510.19055 · HTML: https://mlsanity.com/paper/2510.19055/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Audio-language model reasoning, one of 17 directions in LLM reasoning.
- Builds on 5 of its 46 references that are in the map.
- First posted on arXiv 21 Oct 2025.

## Placed in

- LLM reasoning / [Audio-language model reasoning](https://mlsanity.com/lr/audio-language-model-reasoning/)
- Multimodal LLMs / vision-language models / [Video understanding with MLLMs](https://mlsanity.com/mllm/video-understanding-with-mllms/)

## Builds on (most cited first, 5 of 5)

- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Qwen2.5-Omni Technical Report](https://mlsanity.com/paper/2503.20215/) (2025)
- [Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models](https://mlsanity.com/paper/2507.08128/) (2025)
- [MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix](https://mlsanity.com/paper/2505.13032/) (2025)
- [MMAU-Pro: A Challenging and Comprehensive Benchmark for Holistic Evaluation of Audio General Intelligence](https://mlsanity.com/paper/2508.13992/) (2025)

## Built on by (most cited first, 4 of 4)

- [RAIL: Rethinking Auditory Intelligence in Large Audio-Language Models with a CHC-Grounded Benchmark](https://mlsanity.com/paper/2606.11260/) (2026)
- [Evaluating Multimodal Large Language Models on Core Music Perception Tasks](https://mlsanity.com/paper/2510.22455/) (2025)
- [PitchBench: Measuring Pitch Hearing in Audio-Language Models](https://mlsanity.com/paper/2605.26176/) (2026)
- [AcoustiClaim: A Numeric Claim Benchmark with Instrument Ground Truth](https://mlsanity.com/paper/2609.30483/) (2026)

## Abstract (arXiv)

Multimodal Large Language Models (MLLMs) have demonstrated capabilities in audio understanding, but current evaluations may obscure fundamental weaknesses in relational reasoning. We introduce the Music Understanding and Structural Evaluation (MUSE) Benchmark, an open-source resource with 10 tasks designed to probe fundamental music perception skills. We evaluate four SOTA models (Gemini Pro and Flash, Qwen2.5-Omni, and Audio-Flamingo 3) against a large human baseline (N=200). Our results reveal a wide variance in SOTA capabilities and a persistent gap with human experts. While Gemini Pro succeeds on basic perception, Qwen and Audio Flamingo 3 perform at or near chance, exposing severe perceptual deficits. Furthermore, we find Chain-of-Thought (CoT) prompting provides inconsistent, often detrimental results. Our work provides a critical tool for evaluating invariant musical representations and driving development of more robust AI systems.