mlsanityData to 7 Oct 2026

Aligning Text-to-Music Evaluation with Human Preferences

arXiv 2503.16669 ↗First posted 20 Mar 2025cs.SD cs.AI eess.AScited by 20

On the map: LLM alignment / RLHF / preference optimisation → Preference alignment for recommenders and RAG

Abstract (arXiv)

Despite significant recent advances in generative acoustic text-to-music (TTM) modeling, robust evaluation of these models lags behind, relying in particular on the popular Fr'echet Audio Distance (FAD). In this work, we rigorously study the design space of reference-based divergence metrics for evaluating TTM models through (1) designing four synthetic meta-evaluations to measure sensitivity to particular musical desiderata, and (2) collecting and evaluating on MusicPrefs, the first open-source dataset of human preferences for TTM systems. We find that not only is the standard FAD setup inconsistent on both synthetic and human preference data, but that nearly all existing metrics fail to effectively capture desiderata, and are only weakly correlated with human perception. We propose a new metric, the MAUVE Audio Divergence (MAD), computed on representations from a self-supervised audio embedding model. We find that this metric effectively captures diverse musical desiderata (average rank correlation 0.84 for MAD vs. 0.49 for FAD and also correlates more strongly with MusicPrefs (0.62 vs. 0.14).

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Placed in Preference alignment for recommenders and RAG, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 28 of its 39 references that are in the map. First posted on arXiv 20 Mar 2025.

Builds on

28 of 39 references.

References · 28

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  2. 2017Neural Discrete Representation Learningcited by 4,650
  3. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  4. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 707
  5. 2016CNN Architectures for Large-Scale Audio Classificationcited by 701
  6. 2020Jukebox: A Generative Model for Musiccited by 535
  7. 2023Simple and Controllable Music Generationcited by 480
  8. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 466
  9. 2020Reliable Fidelity and Diversity Metrics for Generative Modelscited by 425
  10. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretrainingcited by 343
  11. 2023Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Modelscited by 323
  12. 2023Rethinking FID: Towards a Better Evaluation Metric for Image Generationcited by 303
16 more
  1. 2016FMA: A Dataset For Music Analysiscited by 205
  2. 2023MusicLDM: Enhancing Novelty in Text-to-Music Generation Using Beat-Synchronous Mixup Strategiescited by 94
  3. 2023Adapting Frechet Audio Distance for Generative Music Evaluationcited by 92
  4. 2024DITTO: Diffusion Inference-Time T-Optimization for Music Generationcited by 73
  5. 2021Codified audio language modeling learns useful representations for music information retrievalcited by 61
  6. 2023Anticipatory Music Transformercited by 35
  7. 2022MAUVE Scores for Generative Models: Theory and Practicecited by 34
  8. 2024Diff-A-Riff: Musical Accompaniment Co-creation via Latent Diffusion Modelscited by 33
  9. 2024DITTO-2: Distilled Diffusion Inference-Time T-Optimization for Music Generationcited by 26
  10. 2025KAD: No More FAD! An Effective and Efficient Evaluation Metric for Audio Generationcited by 24
  11. 2024The VoiceMOS Challenge 2024: Beyond Speech Quality Predictioncited by 17
  12. 2022Evaluating generative audio systems and their metricscited by 16
  13. 2024Presto! Distilling Steps and Layers for Accelerating Music Generationcited by 16
  14. 2024COCOLA: Coherence-Oriented Contrastive Learning of Musical Audio Representationscited by 14
  15. 2024Frechet Music Distance: A Metric For Generative Symbolic Music Evaluationcited by 12
  16. 2024Improving Musical Accompaniment Co-creation via Diffusion Transformerscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 12.

20 papers.

Sort:

By area:

  1. 2025Survey on the Evaluation of Generative Models in Musiccited by 10
  2. 2025Music Arena: Live Evaluation for Text-to-Musiccited by 8
  3. 2026MuQ-Eval: An Open-Source Per-Sample Quality Metric for AI Music Generation Evaluationcited by 6
  4. 2026CMI-RewardBench: Evaluating Music Reward Models with Compositional Multimodal Instructioncited by 5
  5. 2025A Survey on Music Generation from Single-Modal, Cross-Modal, and Multi-Modal Perspectivescited by 4
  6. 2025MGE-LDM: Joint Latent Diffusion for Simultaneous Music Generation and Source Extractioncited by 4
  7. 2026TuneJury: An Open Metric for Improving Music Generation Preference Alignmentcited by 3
  8. 2025Mitigating data replication in text-to-audio generative diffusion models through anti-memorization guidancecited by 2
  9. 2025The Name-Free Gap: Policy-Aware Stylistic Control in Music Generationcited by 1
  10. 2025Do Foundational Audio Encoders Understand Music Structure?cited by 1
  11. 2026ImmersiveFlow: Stereo-to-7.1.4 spatial audio generation with flow matchingcited by 1
  12. 2025AImoclips: A Benchmark for Evaluating Emotion Conveyance in Text-to-Music Generationcited by 0
8 more
  1. 2026When Noise Lowers The Loss: Rethinking Likelihood-Based Evaluation in Music Large Language Modelscited by 0
  2. 2026How Far Can Chord-Symbol Time-Series Adaptation Carry Genre Identity? Capabilities and Boundaries in Multi-Genre Chord-Symbol Modelingcited by 0
  3. 2026Improving Text-to-Music Generation with Human Preference Rewardscited by 0
  4. 2026BackgroundMellow: A Multi-Modal Cohesive Framework for Narrative-Driven Rich Cinematic Soundscape Generationcited by 0
  5. 2026Integrating Contextual Embeddings into Evaluation of Expressive MIDI Piano Performancescited by 0
  6. 2026InvFlowFD: Reference-Free and Background-Set-Free Perceptual Music Quality Metric with Flow Matching Inversioncited by 0
  7. 2026Pitch-class Steering for Diffusion-based Music Generation via Latent-space Probescited by 0
  8. 2026End-to-End Historical Music Restoration in Latent Spacecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Preference-Based Learning in Audio Applications: A Systematic Analysis
  2. 2025Aligning Generative Music AI with Human Preferences: Methods and Challenges
  3. 2025MR-FlowDPO: Multi-Reward Direct Preference Optimization for Flow-Matching Text-to-Music Generation
  4. 2024Tango 2: Aligning Diffusion-based Text-to-Audio Generations through Direct Preference Optimization
  5. 2024MuChoMusic: Evaluating Music Understanding in Multimodal Audio-Language Models
  6. 2023ERNIE-Music: Text-to-Waveform Music Generation with Diffusion Models
  7. 2026Instrumental Text-to-Music Generation with Auxiliary Conditioning Branches ↗
  8. 2025Auto-Regressive vs Flow-Matching: a Comparative Study of Modeling Paradigms for Text-to-Music Generation