# MuChin: A Chinese Colloquial Description Benchmark for Evaluating Language Models in the Field of Music

arXiv: https://arxiv.org/abs/2402.09871 · HTML: https://mlsanity.com/paper/2402.09871/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 44 references that are in the map.
- First posted on arXiv 15 Feb 2024.

## Builds on (most cited first, 29 of 29)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [GLM-130B: An Open Bilingual Pre-trained Model](https://mlsanity.com/paper/2210.02414/) (2022)
- [C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models](https://mlsanity.com/paper/2305.08322/) (2023)
- [High Fidelity Neural Audio Compression](https://mlsanity.com/paper/2210.13438/) (2022)
- [Baichuan 2: Open Large-scale Language Models](https://mlsanity.com/paper/2309.10305/) (2023)
- [Jukebox: A Generative Model for Music](https://mlsanity.com/paper/2005.00341/) (2020)
- [Simple and Controllable Music Generation](https://mlsanity.com/paper/2306.05284/) (2023)
- [MusicLM: Generating Music From Text](https://mlsanity.com/paper/2301.11325/) (2023)
- [MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Training](https://mlsanity.com/paper/2306.00107/) (2023)
- [Noise2Music: Text-conditioned Music Generation with Diffusion Models](https://mlsanity.com/paper/2302.03917/) (2023)
- [Hybrid Transformers for Music Source Separation](https://mlsanity.com/paper/2211.08553/) (2022)
- [Mustango: Toward Controllable Text-to-Music Generation](https://mlsanity.com/paper/2311.08355/) (2023)
- [Hybrid Spectrogram and Waveform Source Separation](https://mlsanity.com/paper/2111.03600/) (2021)
- [MuLan: A Joint Embedding of Music Audio and Natural Language](https://mlsanity.com/paper/2208.12415/) (2022)
- [POP909: A Pop-song Dataset for Music Arrangement Generation](https://mlsanity.com/paper/2008.07142/) (2020)
- [MidiNet: A Convolutional Generative Adversarial Network for Symbolic-domain Music Generation](https://mlsanity.com/paper/1703.10847/) (2017)
- [Codified audio language modeling learns useful representations for music information retrieval](https://mlsanity.com/paper/2107.05677/) (2021)
- [MuseCoco: Generating Symbolic Music from Text](https://mlsanity.com/paper/2306.00110/) (2023)
- [MARBLE: Music Audio Representation Benchmark for Universal Evaluation](https://mlsanity.com/paper/2306.10548/) (2023)
- [MusCaps: Generating Captions for Music Audio](https://mlsanity.com/paper/2104.11984/) (2021)
- [MAP-Music2Vec: A Simple and Effective Baseline for Self-Supervised Music Audio Representation Learning](https://mlsanity.com/paper/2212.02508/) (2022)
- [ERNIE-Music: Text-to-Waveform Music Generation with Diffusion Models](https://mlsanity.com/paper/2302.04456/) (2023)
- [SongDriver: Real-time Music Accompaniment Generation without Logical Latency nor Exposure Bias](https://mlsanity.com/paper/2209.06054/) (2022)
- [SDMuse: Stochastic Differential Music Editing and Generation via Hybrid Representation](https://mlsanity.com/paper/2211.00222/) (2022)
- [REMAST: Real-time Emotion-based Music Arrangement with Soft Transition](https://mlsanity.com/paper/2305.08029/) (2023)

## Built on by (most cited first, 16 of 16)

- [MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark](https://mlsanity.com/paper/2410.19168/) (2024)
- [MuChoMusic: Evaluating Music Understanding in Multimodal Audio-Language Models](https://mlsanity.com/paper/2408.01337/) (2024)
- [CMI-Bench: A Comprehensive Benchmark for Evaluating Music Instruction Following](https://mlsanity.com/paper/2506.12285/) (2025)
- [A Survey on Music Generation from Single-Modal, Cross-Modal, and Multi-Modal Perspectives](https://mlsanity.com/paper/2504.00837/) (2025)
- [Back to Ear: Perceptually Driven High Fidelity Music Reconstruction](https://mlsanity.com/paper/2509.14912/) (2025)
- [Domain Specific Benchmarks for Evaluating Multimodal Large Language Models](https://mlsanity.com/paper/2506.12958/) (2025)
- [MuDiT & MuSiT: Alignment with Colloquial Expression in Description-to-Song Generation](https://mlsanity.com/paper/2407.03188/) (2024)
- [CrossMuSim: A Cross-Modal Framework for Music Similarity Retrieval with LLM-Powered Text Description Sourcing and Mining](https://mlsanity.com/paper/2503.23128/) (2025)
- [Multi-Accent Mandarin Dry-Vocal Singing Dataset: Benchmark for Singing Accent Recognition](https://mlsanity.com/paper/2512.07005/) (2025)
- [Singing Timbre Popularity Assessment Based on Multimodal Large Foundation Model](https://mlsanity.com/paper/2512.06999/) (2025)
- [VocalParse: Towards Unified and Scalable Singing Voice Transcription with Large Audio Language Models](https://mlsanity.com/paper/2605.04613/) (2026)
- [Elastic Time: Dynamic Frame Rate Bottlenecks for Neural Audio Coding](https://mlsanity.com/paper/2606.27320/) (2026)
- [MADB: A Large-Scale Music Aesthetics Dataset with Professional and Multi-Dimensional Annotations](https://mlsanity.com/paper/2607.06929/) (2026)
- [VocalRender: Score-Native Singing Voice Synthesis for Real-World Composition](https://mlsanity.com/paper/2607.27768/) (2026)
- [MusicLayout: Explicit Structural Planning for Controllable Text-to-Music Generation](https://mlsanity.com/paper/2608.09035/) (2026)
- [Open-Qwen-Music: An Auditable Framework for LLM-Based Music Composition and Diffusion Rendering](https://mlsanity.com/paper/2609.31652/) (2026)

## Abstract (arXiv)

The rapidly evolving multimodal Large Language Models (LLMs) urgently require new benchmarks to uniformly evaluate their performance on understanding and textually describing music. However, due to semantic gaps between Music Information Retrieval (MIR) algorithms and human understanding, discrepancies between professionals and the public, and low precision of annotations, existing music description datasets cannot serve as benchmarks. To this end, we present MuChin, the first open-source music description benchmark in Chinese colloquial language, designed to evaluate the performance of multimodal LLMs in understanding and describing music. We established the Caichong Music Annotation Platform (CaiMAP) that employs an innovative multi-person, multi-stage assurance method, and recruited both amateurs and professionals to ensure the precision of annotations and alignment with popular semantics. Utilizing this method, we built a dataset with multi-dimensional, high-precision music annotations, the Caichong Music Dataset (CaiMD), and carefully selected 1,000 high-quality entries to serve as the test set for MuChin. Based on MuChin, we analyzed the discrepancies between professionals and amateurs in terms of music description, and empirically demonstrated the effectiveness of annotated data for fine-tuning LLMs. Ultimately, we employed MuChin to evaluate existing music understanding models on their ability to provide colloquial descriptions of music. All data related to the benchmark, along with the scoring code and detailed appendices, have been open-sourced (https://github.com/CarlWangChina/MuChin/).