# Benchmarking Music Generation Models and Metrics via Human Preference Studies

arXiv: https://arxiv.org/abs/2506.19085 · HTML: https://mlsanity.com/paper/2506.19085/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 26 references that are in the map.
- First posted on arXiv 23 Jun 2025.

## Builds on (most cited first, 16 of 16)

- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [High Fidelity Neural Audio Compression](https://mlsanity.com/paper/2210.13438/) (2022)
- [PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition](https://mlsanity.com/paper/1912.10211/) (2019)
- [Simple and Controllable Music Generation](https://mlsanity.com/paper/2306.05284/) (2023)
- [MusicLM: Generating Music From Text](https://mlsanity.com/paper/2301.11325/) (2023)
- [AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining](https://mlsanity.com/paper/2308.05734/) (2023)
- [Fr'echet Audio Distance: A Metric for Evaluating Music Enhancement Algorithms](https://mlsanity.com/paper/1812.08466/) (2018)
- [FMA: A Dataset For Music Analysis](https://mlsanity.com/paper/1612.01840/) (2016)
- [Fast Timing-Conditioned Latent Audio Diffusion](https://mlsanity.com/paper/2402.04825/) (2024)
- [Noise2Music: Text-conditioned Music Generation with Diffusion Models](https://mlsanity.com/paper/2302.03917/) (2023)
- [Mustango: Toward Controllable Text-to-Music Generation](https://mlsanity.com/paper/2311.08355/) (2023)
- [Adapting Frechet Audio Distance for Generative Music Evaluation](https://mlsanity.com/paper/2311.01616/) (2023)
- [Natural Language Supervision for General-Purpose Audio Representations](https://mlsanity.com/paper/2309.05767/) (2023)
- [Evaluating generative audio systems and their metrics](https://mlsanity.com/paper/2209.00130/) (2022)

## Built on by (most cited first, 15 of 15)

- [Music Arena: Live Evaluation for Text-to-Music](https://mlsanity.com/paper/2507.20900/) (2025)
- [MuQ-Eval: An Open-Source Per-Sample Quality Metric for AI Music Generation Evaluation](https://mlsanity.com/paper/2603.22677/) (2026)
- [CMI-RewardBench: Evaluating Music Reward Models with Compositional Multimodal Instruction](https://mlsanity.com/paper/2603.00610/) (2026)
- [TuneJury: An Open Metric for Improving Music Generation Preference Alignment](https://mlsanity.com/paper/2606.17006/) (2026)
- [Bias beyond Borders: Global Inequalities in AI-Generated Music](https://mlsanity.com/paper/2510.01963/) (2025)
- [Fusion Segment Transformer: Bi-Directional Attention Guided Fusion Network for AI-Generated Music Detection](https://mlsanity.com/paper/2601.13647/) (2026)
- [Addressing Performance Saturation for LLM RL via Precise Entropy Curve Control](https://mlsanity.com/paper/2604.26326/) (2026)
- [AImoclips: A Benchmark for Evaluating Emotion Conveyance in Text-to-Music Generation](https://mlsanity.com/paper/2509.00813/) (2025)
- [Echoes of Humanity: Exploring the Perceived Humanness of AI Music](https://mlsanity.com/paper/2509.25601/) (2025)
- [When Noise Lowers The Loss: Rethinking Likelihood-Based Evaluation in Music Large Language Models](https://mlsanity.com/paper/2602.02738/) (2026)
- [Echoes: A semantically-aligned music deepfake detection dataset](https://mlsanity.com/paper/2603.23667/) (2026)
- [Seconds-Aligned PCA-DAC Latent Diffusion for Symbolic-to-Audio Drum Rendering](https://mlsanity.com/paper/2605.13404/) (2026)
- [HAIM: Human-AI Music Datasets for AI Music Production Tracking Benchmark](https://mlsanity.com/paper/2606.01686/) (2026)
- [Improving Text-to-Music Generation with Human Preference Rewards](https://mlsanity.com/paper/2606.21670/) (2026)
- [TTM-Bench: A Framework for Text-to-Music System Performance Benchmarking](https://mlsanity.com/paper/2609.18585/) (2026)

## Abstract (arXiv)

Recent advancements have brought generated music closer to human-created compositions, yet evaluating these models remains challenging. While human preference is the gold standard for assessing quality, translating these subjective judgments into objective metrics, particularly for text-audio alignment and music quality, has proven difficult. In this work, we generate 6k songs using 12 state-of-the-art models and conduct a survey of 15k pairwise audio comparisons with 2.5k human participants to evaluate the correlation between human preferences and widely used metrics. To the best of our knowledge, this work is the first to rank current state-of-the-art music generation models and metrics based on human preference. To further the field of subjective metric evaluation, we provide open access to our dataset of generated music and human evaluations.