# CMI-Bench: A Comprehensive Benchmark for Evaluating Music Instruction Following

arXiv: https://arxiv.org/abs/2506.12285 · HTML: https://mlsanity.com/paper/2506.12285/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 74 references that are in the map.
- First posted on arXiv 14 Jun 2025.

## Builds on (most cited first, 30 of 44)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks](https://mlsanity.com/paper/2204.07705/) (2022)
- [Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models](https://mlsanity.com/paper/2311.07919/) (2023)
- [C-Pack: Packed Resources For General Chinese Embeddings](https://mlsanity.com/paper/2309.07597/) (2023)
- [Qwen2-Audio Technical Report](https://mlsanity.com/paper/2407.10759/) (2024)
- [SALMONN: Towards Generic Hearing Abilities for Large Language Models](https://mlsanity.com/paper/2310.13289/) (2023)
- [MusicLM: Generating Music From Text](https://mlsanity.com/paper/2301.11325/) (2023)
- [MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark](https://mlsanity.com/paper/2410.19168/) (2024)
- [Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders](https://mlsanity.com/paper/1704.01279/) (2017)
- [Pengi: An Audio Language Model for Audio Tasks](https://mlsanity.com/paper/2305.11834/) (2023)
- [AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehension](https://mlsanity.com/paper/2402.07729/) (2024)
- [Listen, Think, and Understand](https://mlsanity.com/paper/2305.10790/) (2023)
- [Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilities](https://mlsanity.com/paper/2402.01831/) (2024)
- [MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Training](https://mlsanity.com/paper/2306.00107/) (2023)
- [GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilities](https://mlsanity.com/paper/2406.11768/) (2024)
- [Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilities](https://mlsanity.com/paper/2503.03983/) (2025)
- [AudioBench: A Universal Benchmark for Audio Large Language Models](https://mlsanity.com/paper/2406.16020/) (2024)
- [OmniBench: Towards The Future of Universal Omni-Language Models](https://mlsanity.com/paper/2409.15272/) (2024)
- [Joint Audio and Speech Understanding](https://mlsanity.com/paper/2309.14405/) (2023)
- [Mustango: Toward Controllable Text-to-Music Generation](https://mlsanity.com/paper/2311.08355/) (2023)
- [MuLan: A Joint Embedding of Music Audio and Natural Language](https://mlsanity.com/paper/2208.12415/) (2022)
- [LP-MusicCaps: LLM-Based Pseudo Music Captioning](https://mlsanity.com/paper/2307.16372/) (2023)
- [Music Understanding LLaMA: Advancing Text-to-Music Generation with Question Answering and Captioning](https://mlsanity.com/paper/2308.11276/) (2023)
- [The Song Describer Dataset: a Corpus of Audio Captions for Music-and-Language Evaluation](https://mlsanity.com/paper/2311.10057/) (2023)
- [Codified audio language modeling learns useful representations for music information retrieval](https://mlsanity.com/paper/2107.05677/) (2021)
- [Towards Learning Universal Audio Representations](https://mlsanity.com/paper/2111.12124/) (2021)
- [MusiLingo: Bridging Music and Text with Pre-trained Language Models for Music Captioning and Query Response](https://mlsanity.com/paper/2309.08730/) (2023)

## Built on by (most cited first, 17 of 17)

- [ABC-Eval: Benchmarking Large Language Models on Symbolic Music Understanding and Instruction Following](https://mlsanity.com/paper/2509.23350/) (2025)
- [The Interspeech 2026 Audio Reasoning Challenge: Evaluating Reasoning Process Quality for Audio Reasoning Models and Agents](https://mlsanity.com/paper/2602.14224/) (2026)
- [ISA-Bench: Benchmarking Instruction Sensitivity for Large Audio Language Models](https://mlsanity.com/paper/2510.23558/) (2025)
- [Echo: Towards Advanced Audio Comprehension via Audio-Interleaved Reasoning](https://mlsanity.com/paper/2602.11909/) (2026)
- [CSyMR: Benchmarking Compositional Music Information Retrieval in Symbolic Music Reasoning](https://mlsanity.com/paper/2601.11556/) (2025)
- [Audio-Visual Intelligence in Large Foundation Models](https://mlsanity.com/paper/2605.04045/) (2026)
- [Evaluating Multimodal Large Language Models on Core Music Perception Tasks](https://mlsanity.com/paper/2510.22455/) (2025)
- [When Noise Lowers The Loss: Rethinking Likelihood-Based Evaluation in Music Large Language Models](https://mlsanity.com/paper/2602.02738/) (2026)
- [BASS: Benchmarking Audio LMs for Musical Structure and Semantic Reasoning](https://mlsanity.com/paper/2602.04085/) (2026)
- [PitchBench: Measuring Pitch Hearing in Audio-Language Models](https://mlsanity.com/paper/2605.26176/) (2026)
- [Exploring LLMs for South Asian Music Understanding and Generation](https://mlsanity.com/paper/2606.05522/) (2026)
- [AudioDER: A Deduplication-Enhanced Reasoning Dataset for Post-Training Large Audio-Language Models](https://mlsanity.com/paper/2606.14591/) (2026)
- [MIDI-RAE-JEPA: Hierarchical Representation Learning and Generation for Symbolic Music](https://mlsanity.com/paper/2607.14537/) (2026)
- [SEA-SpeechBench: A Large-Scale Multitask Benchmark for Speech Understanding Across Southeast Asia](https://mlsanity.com/paper/2609.09672/) (2026)
- [MUUNRiver-Bench: Diagnosing Relation-Dependent Music Retrieval with Multimodal Instructions](https://mlsanity.com/paper/2609.16090/) (2026)
- [Music Hallucination in Audio-Language Models: A Hierarchical Formulation and Empirical Study](https://mlsanity.com/paper/2609.20195/) (2026)
- [Extending Music Annotation Schemas: Zero-Shot Prediction or Few-Shot Adaptation?](https://mlsanity.com/paper/2610.06920/) (2026)

## Abstract (arXiv)

Recent advances in audio-text large language models (LLMs) have opened new possibilities for music understanding and generation. However, existing benchmarks are limited in scope, often relying on simplified tasks or multi-choice evaluations that fail to reflect the complexity of real-world music analysis. We reinterpret a broad range of traditional MIR annotations as instruction-following formats and introduce CMI-Bench, a comprehensive music instruction following benchmark designed to evaluate audio-text LLMs on a diverse set of music information retrieval (MIR) tasks. These include genre classification, emotion regression, emotion tagging, instrument classification, pitch estimation, key detection, lyrics transcription, melody extraction, vocal technique recognition, instrument performance technique detection, music tagging, music captioning, and (down)beat tracking: reflecting core challenges in MIR research. Unlike previous benchmarks, CMI-Bench adopts standardized evaluation metrics consistent with previous state-of-the-art MIR models, ensuring direct comparability with supervised approaches. We provide an evaluation toolkit supporting all open-source audio-textual LLMs, including LTU, Qwen-audio, SALMONN, MusiLingo, etc. Experiment results reveal significant performance gaps between LLMs and supervised models, along with their culture, chronological and gender bias, highlighting the potential and limitations of current models in addressing MIR tasks. CMI-Bench establishes a unified foundation for evaluating music instruction following, driving progress in music-aware LLMs.