# MusicEval: A Generative Music Dataset with Expert Ratings for Automatic Text-to-Music Evaluation

arXiv: https://arxiv.org/abs/2501.10811 · HTML: https://mlsanity.com/paper/2501.10811/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Jan 2025.

## Built on by (most cited first, 18 of 18)

- [Survey on the Evaluation of Generative Models in Music](https://mlsanity.com/paper/2506.05104/) (2025)
- [Music Arena: Live Evaluation for Text-to-Music](https://mlsanity.com/paper/2507.20900/) (2025)
- [AQAScore: Evaluating Semantic Alignment in Text-to-Audio Generation via Audio Question Answering](https://mlsanity.com/paper/2601.14728/) (2026)
- [SongBench: A Fine-Grained Multi-Aspect Benchmark for Song Quality Assessment](https://mlsanity.com/paper/2604.25937/) (2026)
- [MuQ-Eval: An Open-Source Per-Sample Quality Metric for AI Music Generation Evaluation](https://mlsanity.com/paper/2603.22677/) (2026)
- [CMI-RewardBench: Evaluating Music Reward Models with Compositional Multimodal Instruction](https://mlsanity.com/paper/2603.00610/) (2026)
- [TuneJury: An Open Metric for Improving Music Generation Preference Alignment](https://mlsanity.com/paper/2606.17006/) (2026)
- [QAMRO: Quality-aware Adaptive Margin Ranking Optimization for Human-aligned Assessment of Audio Generation Systems](https://mlsanity.com/paper/2508.08957/) (2025)
- [WhisQ: Cross-Modal Representation Learning for Text-to-Music MOS Prediction](https://mlsanity.com/paper/2506.05899/) (2025)
- [DRASP: A Dual-Resolution Attentive Statistics Pooling Framework for Automatic MOS Prediction](https://mlsanity.com/paper/2508.21407/) (2025)
- [AImoclips: A Benchmark for Evaluating Emotion Conveyance in Text-to-Music Generation](https://mlsanity.com/paper/2509.00813/) (2025)
- [Art2Mus: Artwork-to-Music Generation via Visual Conditioning and Large-Scale Cross-Modal Alignment](https://mlsanity.com/paper/2602.17599/) (2026)
- [LabelBuddy: An Open Source Music and Audio Language Annotation Tagging Tool Using AI Assistance](https://mlsanity.com/paper/2603.04293/) (2026)
- [DeRA-MOS: Optimizing Text-to-Music Evaluation via Decoupled Listwise Ranking and Modality Alignment](https://mlsanity.com/paper/2606.10010/) (2026)
- [MADB: A Large-Scale Music Aesthetics Dataset with Professional and Multi-Dimensional Annotations](https://mlsanity.com/paper/2607.06929/) (2026)
- [TTM-Bench: A Framework for Text-to-Music System Performance Benchmarking](https://mlsanity.com/paper/2609.18585/) (2026)
- [MIRA: A Musical Intent Refinement Agent for Aligning Text-to-Music Generation with User Intent](https://mlsanity.com/paper/2610.10355/) (2026)
- [What Do Audio-Language Judges of Generated Music Listen For? Four Probes with Controls from Expert Ratings](https://mlsanity.com/paper/2610.13247/) (2026)

## Abstract (arXiv)

The technology for generating music from textual descriptions has seen rapid advancements. However, evaluating text-to-music (TTM) systems remains a significant challenge, primarily due to the difficulty of balancing performance and cost with existing objective and subjective evaluation methods. In this paper, we propose an automatic assessment task for TTM models to align with human perception. To address the TTM evaluation challenges posed by the professional requirements of music evaluation and the complexity of the relationship between text and music, we collect MusicEval, the first generative music assessment dataset. This dataset contains 2,748 music clips generated by 31 advanced and widely used models in response to 384 text prompts, along with 13,740 ratings from 14 music experts. Furthermore, we design a CLAP-based assessment model built on this dataset, and our experimental results validate the feasibility of the proposed task, providing a valuable reference for future development in TTM evaluation. The dataset is available at https://www.aishelltech.com/AISHELL_7A.