mlsanityData to 14 Oct 2026

Aligning Generative Music AI with Human Preferences: Methods and Challenges

arXiv 2511.15038 ↗First posted 19 Nov 2025cs.SD cs.AI eess.AScited by 0

On the map: LLM alignment / RLHF / preference optimisation → Preference alignment for recommenders and RAG

Abstract (arXiv)

Recent advances in generative AI for music have achieved remarkable fidelity and stylistic diversity, yet these systems often fail to align with nuanced human preferences due to the specific loss functions they use. This paper advocates for the systematic application of preference alignment techniques to music generation, addressing the fundamental gap between computational optimization and human musical appreciation. Drawing on recent breakthroughs including MusicRL's large-scale preference learning, multi-preference alignment frameworks like diffusion-based preference optimization in DiffRhythm+, and inference-time optimization techniques like Text2midi-InferAlign, we discuss how these techniques can address music's unique challenges: temporal coherence, harmonic consistency, and subjective quality assessment. We identify key research challenges including scalability to long-form compositions, reliability amongst others in preference modelling. Looking forward, we envision preference-aligned music generation enabling transformative applications in interactive composition tools and personalized music services. This work calls for sustained interdisciplinary research combining advances in machine learning, music-theory to create music AI systems that truly serve human creative and experiential needs.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Preference alignment for recommenders and RAG, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 24 of its 41 references that are in the map. First posted on arXiv 19 Nov 2025.

Builds on

24 of 41 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024MusicRL: Aligning Music Generation to Human Preferences6 Feb

Other references · 23

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2017Deep reinforcement learning from human preferencescited by 3,912
  4. 2016Improved Techniques for Training GANscited by 2,970
  5. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  6. 2017Mixed Precision Trainingcited by 1,156
  7. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  8. 2020Jukebox: A Generative Model for Musiccited by 542
  9. 2018A Note on the Inception Scorecited by 226
  10. 2018Fr'echet Audio Distance: A Metric for Evaluating Music Enhancement Algorithmscited by 225
  11. 2025Meta Audiobox Aesthetics: Unified Automatic Quality Assessment for Speech, Music, and Soundcited by 137
  12. 2023Mustango: Toward Controllable Text-to-Music Generationcited by 100
11 more
  1. 2024DITTO: Diffusion Inference-Time T-Optimization for Music Generationcited by 73
  2. 2023Video2Music: Suitable Music Generation from Videos using an Affective Multimodal Transformer modelcited by 37
  3. 2018A Functional Taxonomy of Music Generation Systemscited by 31
  4. 2025NotaGen: Advancing Musicality in Symbolic Music Generation with Large Language Model Training Paradigmscited by 30
  5. 2025SongEval: A Benchmark Dataset for Song Aesthetics Evaluationcited by 26
  6. 2024Text2midi: Generating Symbolic Music from Captionscited by 19
  7. 2025JamendoMaxCaps: A Large Scale Music-caption Dataset with Imputed Metadatacited by 15
  8. 2021Hierarchical Recurrent Neural Networks for Conditional Melody Generation with Long-term Structurecited by 12
  9. 2025DiffRhythm+: Controllable and Flexible Full-Length Song Generation with Preference Optimizationcited by 12
  10. 2024SMITIN: Self-Monitored Inference-Time INtervention for Generative Music Transformerscited by 9
  11. 2025Text2midi-InferAlign: Improving Symbolic Music Generation with Inference-Time Alignmentcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026TuneJury: An Open Metric for Improving Music Generation Preference Alignment
  2. 2025MR-FlowDPO: Multi-Reward Direct Preference Optimization for Flow-Matching Text-to-Music Generation
  3. 2025TuneGenie: Reasoning-based LLM agents for preferential music generation
  4. 2026Aligning Language Models for Lyric-to-Melody Generation with Rule-Based Musical Constraints
  5. 2025Aligning Text-to-Music Evaluation with Human Preferences
  6. 2025LeVo: High-Quality Song Generation with Multi-Preference Alignment
  7. 2025Preference-Based Learning in Audio Applications: A Systematic Analysis
  8. 2026Improving Text-to-Music Generation with Human Preference Rewards