# Aligning Text-to-Music Evaluation with Human Preferences

arXiv: https://arxiv.org/abs/2503.16669 · HTML: https://mlsanity.com/paper/2503.16669/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Placed in Preference alignment for recommenders and RAG, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 28 of its 39 references that are in the map.
- First posted on arXiv 20 Mar 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Preference alignment for recommenders and RAG](https://mlsanity.com/alignment/preference-alignment-for-recommenders-and-rag/)

## Builds on (most cited first, 28 of 28)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [Jukebox: A Generative Model for Music](https://mlsanity.com/paper/2005.00341/) (2020)
- [Simple and Controllable Music Generation](https://mlsanity.com/paper/2306.05284/) (2023)
- [MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers](https://mlsanity.com/paper/2102.01454/) (2021)
- [Reliable Fidelity and Diversity Metrics for Generative Models](https://mlsanity.com/paper/2002.09797/) (2020)
- [AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining](https://mlsanity.com/paper/2308.05734/) (2023)
- [Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models](https://mlsanity.com/paper/2301.12661/) (2023)
- [Rethinking FID: Towards a Better Evaluation Metric for Image Generation](https://mlsanity.com/paper/2401.09603/) (2023)
- [FMA: A Dataset For Music Analysis](https://mlsanity.com/paper/1612.01840/) (2016)
- [MusicLDM: Enhancing Novelty in Text-to-Music Generation Using Beat-Synchronous Mixup Strategies](https://mlsanity.com/paper/2308.01546/) (2023)
- [Adapting Frechet Audio Distance for Generative Music Evaluation](https://mlsanity.com/paper/2311.01616/) (2023)
- [DITTO: Diffusion Inference-Time T-Optimization for Music Generation](https://mlsanity.com/paper/2401.12179/) (2024)
- [Codified audio language modeling learns useful representations for music information retrieval](https://mlsanity.com/paper/2107.05677/) (2021)
- [MAUVE Scores for Generative Models: Theory and Practice](https://mlsanity.com/paper/2212.14578/) (2022)
- [Anticipatory Music Transformer](https://mlsanity.com/paper/2306.08620/) (2023)
- [Diff-A-Riff: Musical Accompaniment Co-creation via Latent Diffusion Models](https://mlsanity.com/paper/2406.08384/) (2024)
- [DITTO-2: Distilled Diffusion Inference-Time T-Optimization for Music Generation](https://mlsanity.com/paper/2405.20289/) (2024)
- [KAD: No More FAD! An Effective and Efficient Evaluation Metric for Audio Generation](https://mlsanity.com/paper/2502.15602/) (2025)
- [The VoiceMOS Challenge 2024: Beyond Speech Quality Prediction](https://mlsanity.com/paper/2409.07001/) (2024)
- [Evaluating generative audio systems and their metrics](https://mlsanity.com/paper/2209.00130/) (2022)
- [Presto! Distilling Steps and Layers for Accelerating Music Generation](https://mlsanity.com/paper/2410.05167/) (2024)
- [Frechet Music Distance: A Metric For Generative Symbolic Music Evaluation](https://mlsanity.com/paper/2412.07948/) (2024)
- [COCOLA: Coherence-Oriented Contrastive Learning of Musical Audio Representations](https://mlsanity.com/paper/2404.16969/) (2024)
- [Improving Musical Accompaniment Co-creation via Diffusion Transformers](https://mlsanity.com/paper/2410.23005/) (2024)

## Built on by (most cited first, 21 of 21)

- [Survey on the Evaluation of Generative Models in Music](https://mlsanity.com/paper/2506.05104/) (2025)
- [Music Arena: Live Evaluation for Text-to-Music](https://mlsanity.com/paper/2507.20900/) (2025)
- [MuQ-Eval: An Open-Source Per-Sample Quality Metric for AI Music Generation Evaluation](https://mlsanity.com/paper/2603.22677/) (2026)
- [CMI-RewardBench: Evaluating Music Reward Models with Compositional Multimodal Instruction](https://mlsanity.com/paper/2603.00610/) (2026)
- [A Survey on Music Generation from Single-Modal, Cross-Modal, and Multi-Modal Perspectives](https://mlsanity.com/paper/2504.00837/) (2025)
- [MGE-LDM: Joint Latent Diffusion for Simultaneous Music Generation and Source Extraction](https://mlsanity.com/paper/2505.23305/) (2025)
- [TuneJury: An Open Metric for Improving Music Generation Preference Alignment](https://mlsanity.com/paper/2606.17006/) (2026)
- [Mitigating data replication in text-to-audio generative diffusion models through anti-memorization guidance](https://mlsanity.com/paper/2509.14934/) (2025)
- [The Name-Free Gap: Policy-Aware Stylistic Control in Music Generation](https://mlsanity.com/paper/2509.00654/) (2025)
- [Do Foundational Audio Encoders Understand Music Structure?](https://mlsanity.com/paper/2512.17209/) (2025)
- [ImmersiveFlow: Stereo-to-7.1.4 spatial audio generation with flow matching](https://mlsanity.com/paper/2601.12950/) (2026)
- [AImoclips: A Benchmark for Evaluating Emotion Conveyance in Text-to-Music Generation](https://mlsanity.com/paper/2509.00813/) (2025)
- [When Noise Lowers The Loss: Rethinking Likelihood-Based Evaluation in Music Large Language Models](https://mlsanity.com/paper/2602.02738/) (2026)
- [How Far Can Chord-Symbol Time-Series Adaptation Carry Genre Identity? Capabilities and Boundaries in Multi-Genre Chord-Symbol Modeling](https://mlsanity.com/paper/2606.07334/) (2026)
- [Improving Text-to-Music Generation with Human Preference Rewards](https://mlsanity.com/paper/2606.21670/) (2026)
- [BackgroundMellow: A Multi-Modal Cohesive Framework for Narrative-Driven Rich Cinematic Soundscape Generation](https://mlsanity.com/paper/2607.11364/) (2026)
- [Integrating Contextual Embeddings into Evaluation of Expressive MIDI Piano Performances](https://mlsanity.com/paper/2607.27909/) (2026)
- [InvFlowFD: Reference-Free and Background-Set-Free Perceptual Music Quality Metric with Flow Matching Inversion](https://mlsanity.com/paper/2608.04142/) (2026)
- [Pitch-class Steering for Diffusion-based Music Generation via Latent-space Probes](https://mlsanity.com/paper/2609.04516/) (2026)
- [End-to-End Historical Music Restoration in Latent Space](https://mlsanity.com/paper/2610.00607/) (2026)
- [What Do Audio-Language Judges of Generated Music Listen For? Four Probes with Controls from Expert Ratings](https://mlsanity.com/paper/2610.13247/) (2026)

## Abstract (arXiv)

Despite significant recent advances in generative acoustic text-to-music (TTM) modeling, robust evaluation of these models lags behind, relying in particular on the popular Fr'echet Audio Distance (FAD). In this work, we rigorously study the design space of reference-based divergence metrics for evaluating TTM models through (1) designing four synthetic meta-evaluations to measure sensitivity to particular musical desiderata, and (2) collecting and evaluating on MusicPrefs, the first open-source dataset of human preferences for TTM systems. We find that not only is the standard FAD setup inconsistent on both synthetic and human preference data, but that nearly all existing metrics fail to effectively capture desiderata, and are only weakly correlated with human perception. We propose a new metric, the MAUVE Audio Divergence (MAD), computed on representations from a self-supervised audio embedding model. We find that this metric effectively captures diverse musical desiderata (average rank correlation 0.84 for MAD vs. 0.49 for FAD and also correlates more strongly with MusicPrefs (0.62 vs. 0.14).