# BASS: Benchmarking Audio LMs for Musical Structure and Semantic Reasoning

arXiv: https://arxiv.org/abs/2602.04085 · HTML: https://mlsanity.com/paper/2602.04085/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Audio-language model reasoning, one of 17 directions in LLM reasoning.
- Builds on 23 of its 51 references that are in the map.
- First posted on arXiv 3 Feb 2026.

## Placed in

- LLM reasoning / [Audio-language model reasoning](https://mlsanity.com/lr/audio-language-model-reasoning/)

## Builds on (most cited first, 23 of 23)

- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [Gemini: A Family of Highly Capable Multimodal Models](https://mlsanity.com/paper/2312.11805/) (2023)
- [Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs](https://mlsanity.com/paper/2503.01743/) (2025)
- [SALMONN: Towards Generic Hearing Abilities for Large Language Models](https://mlsanity.com/paper/2310.13289/) (2023)
- [Qwen3-Omni Technical Report](https://mlsanity.com/paper/2509.17765/) (2025)
- [MusicLM: Generating Music From Text](https://mlsanity.com/paper/2301.11325/) (2023)
- [Kimi-Audio Technical Report](https://mlsanity.com/paper/2504.18425/) (2025)
- [AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehension](https://mlsanity.com/paper/2402.07729/) (2024)
- [Audio-Reasoner: Improving Reasoning Capability in Large Audio Language Models](https://mlsanity.com/paper/2503.02318/) (2025)
- [Music Understanding LLaMA: Advancing Text-to-Music Generation with Question Answering and Captioning](https://mlsanity.com/paper/2308.11276/) (2023)
- [Reinforcement Learning Outperforms Supervised Fine-Tuning: A Case Study on Audio Question Answering](https://mlsanity.com/paper/2503.11197/) (2025)
- [MMAU-Pro: A Challenging and Comprehensive Benchmark for Holistic Evaluation of Audio General Intelligence](https://mlsanity.com/paper/2508.13992/) (2025)
- [Voxtral](https://mlsanity.com/paper/2507.13264/) (2025)
- [Music Source Separation with Band-Split RoPE Transformer](https://mlsanity.com/paper/2309.02612/) (2023)
- [Step-Audio-R1 Technical Report](https://mlsanity.com/paper/2511.15848/) (2025)
- [Mellow: a small audio language model for reasoning](https://mlsanity.com/paper/2503.08540/) (2025)
- [Music Flamingo: Scaling Music Understanding in Audio Language Models](https://mlsanity.com/paper/2511.10289/) (2025)
- [CMI-Bench: A Comprehensive Benchmark for Evaluating Music Instruction Following](https://mlsanity.com/paper/2506.12285/) (2025)
- [BLAB: Brutally Long Audio Bench](https://mlsanity.com/paper/2505.03054/) (2025)
- [AudSemThinker: Enhancing Audio-Language Models through Reasoning over Semantics of Sound](https://mlsanity.com/paper/2505.14142/) (2025)
- [Lyrics Transcription for Humans: A Readability-Aware Benchmark](https://mlsanity.com/paper/2408.06370/) (2024)
- [Towards Playlist Generation Algorithms Using RNNs Trained on Within-Track Transitions](https://mlsanity.com/paper/1606.02096/) (2016)
- [Exploiting Music Source Separation for Automatic Lyrics Transcription with Whisper](https://mlsanity.com/paper/2506.15514/) (2025)

## Abstract (arXiv)

Music understanding is a complex task that often requires reasoning over both structural and semantic elements of audio. We introduce BASS, designed to evaluate music understanding and reasoning in audio language models across four broad categories: structural segmentation, lyric transcription, musicological analysis, and artist collaboration. BASS comprises 2658 questions spanning 12 tasks, 1993 unique songs and covering over 138 hours of music from a wide range of genres and tracks, crafted to assess musicological knowledge and reasoning in real-world scenarios. We evaluate 14 open-source and frontier multimodal LMs, finding that even state-of-the-art models struggle on higher-level reasoning tasks such as structural segmentation and artist collaboration, while performing best on lyric transcription. Our analysis reveals that current models leverage linguistic priors effectively but remain limited in reasoning over musical structure, vocal, and musicological attributes. BASS provides an evaluation framework with widespread applications in music recommendation and search and has the potential to guide the development of audio LMs.