mlsanityData to 7 Oct 2026

MusicRL: Aligning Music Generation to Human Preferences

arXiv 2402.04229 ↗First posted 6 Feb 2024cs.LG cs.SD eess.AScited by 32

On the map: LLM alignment / RLHF / preference optimisation → Preference alignment for recommenders and RAG

Abstract (arXiv)

We propose MusicRL, the first music generation system finetuned from human feedback. Appreciation of text-to-music models is particularly subjective since the concept of musicality as well as the specific intention behind a caption are user-dependent (e.g. a caption such as "upbeat work-out music" can map to a retro guitar solo or a techno pop beat). Not only this makes supervised training of such models challenging, but it also calls for integrating continuous human feedback in their post-deployment finetuning. MusicRL is a pretrained autoregressive MusicLM (Agostinelli et al., 2023) model of discrete audio tokens finetuned with reinforcement learning to maximise sequence-level rewards. We design reward functions related specifically to text-adherence and audio quality with the help from selected raters, and use those to finetune MusicLM into MusicRL-R. We deploy MusicLM to users and collect a substantial dataset comprising 300,000 pairwise preferences. Using Reinforcement Learning from Human Feedback (RLHF), we train MusicRL-U, the first text-to-music model that incorporates human feedback at scale. Human evaluations show that both MusicRL-R and MusicRL-U are preferred to the baseline. Ultimately, MusicRL-RU combines the two approaches and results in the best model according to human raters. Ablation studies shed light on the musical attributes influencing human preferences, indicating that text adherence and quality only account for a part of it. This underscores the prevalence of subjectivity in musical appreciation and calls for further involvement of human listeners in the finetuning of music generation models.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Placed in Preference alignment for recommenders and RAG, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 40 of its 57 references that are in the map. First posted on arXiv 6 Feb 2024.

Builds on

40 of 57 references.

References · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  8. 2020Learning to summarize from human feedbackcited by 2,562
  9. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  10. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  11. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  12. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 767
28 more
  1. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  2. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  3. 2022Autoregressive Image Generation using Residual Quantizationcited by 706
  4. 2022High Fidelity Neural Audio Compressioncited by 673
  5. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  6. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Modelscited by 592
  7. 2015Sequence Level Training with Recurrent Neural Networkscited by 568
  8. 2020Jukebox: A Generative Model for Musiccited by 535
  9. 2023Aligning Text-to-Image Models using Human Feedbackcited by 418
  10. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  11. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  12. 2022Diffsound: Discrete Diffusion Model for Text-to-sound Generationcited by 256
  13. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 225
  14. 2017Neural Audio Synthesis of Musical Notes with WaveNet Autoencoderscited by 222
  15. 2020DDSP: Differentiable Digital Signal Processingcited by 177
  16. 2023Noise2Music: Text-conditioned Music Generation with Diffusion Modelscited by 173
  17. 2023Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervisioncited by 167
  18. 2023MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformerscited by 164
  19. 2017Objective-Reinforced Generative Adversarial Networks (ORGAN) for Sequence Generation Modelscited by 116
  20. 2016Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-controlcited by 100
  21. 2022MuLan: A Joint Embedding of Music Audio and Natural Languagecited by 85
  22. 2023Factually Consistent Summarization via Reinforcement Learning with Textual Entailment Feedbackcited by 71
  23. 2018Learning to Extract Coherent Summary via Deep Reinforcement Learningcited by 63
  24. 2022General-purpose, long-context autoregressive modeling with Perceiver ARcited by 60
  25. 2018SING: Symbol-to-Instrument Neural Generatorcited by 23
  26. 2020RL-Duet: Online Music Accompaniment Generation Using Deep Reinforcement Learningcited by 19
  27. 2021A Survey on Deep Reinforcement Learning for Audio-Based Applicationscited by 17
  28. 2018Bach2Bach: Generating Music Using A Deep Reinforcement Learning Approachcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 17; 2026: 8.

32 papers.

Sort:

By area:

  1. 2024Long-form music generation with latent diffusioncited by 88
  2. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 88
  3. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  4. 2025NotaGen: Advancing Musicality in Symbolic Music Generation with Large Language Model Training Paradigmscited by 29
  5. 2025LeVo: High-Quality Song Generation with Multi-Preference Alignmentcited by 23
  6. 2024Enhancing Zero-shot Text-to-Speech Synthesis with Human Feedbackcited by 19
  7. 2025Aligning Spoken Dialogue Models from User Interactionscited by 11
  8. 2024Diversity-Rewarded CFG Distillationcited by 9
  9. 2025Music Arena: Live Evaluation for Text-to-Musiccited by 8
  10. 2025Multi-Metric Preference Alignment for Generative Speech Restorationcited by 7
  11. 2025Vevo2: A Unified and Controllable Framework for Speech and Singing Voice Generationcited by 5
  12. 2025MR-FlowDPO: Multi-Reward Direct Preference Optimization for Flow-Matching Text-to-Music Generationcited by 5
20 more
  1. 2026CMI-RewardBench: Evaluating Music Reward Models with Compositional Multimodal Instructioncited by 5
  2. 2025DRAGON: Distributional Rewards Optimize Diffusion Generative Modelscited by 4
  3. 2025Text2midi-InferAlign: Improving Symbolic Music Generation with Inference-Time Alignmentcited by 4
  4. 2025SpectroStream: A Versatile Neural Codec for General Audiocited by 4
  5. 2026LeVo 2: Stable and Melodious Song Generation via Hierarchical Representation Modeling and Progressive Post-Trainingcited by 4
  6. 2024Dynamic Normativity: Necessary and Sufficient Conditions for Value Alignmentcited by 3
  7. 2024Bailing-TTS: Chinese Dialectal Speech Synthesis Towards Human-like Spontaneous Representationcited by 3
  8. 2026TuneJury: An Open Metric for Improving Music Generation Preference Alignmentcited by 3
  9. 2024MAD Speech: Measures of Acoustic Diversity of Speechcited by 2
  10. 2025Toward Rich Video Human-Motion2D Generationcited by 2
  11. 2025Towards Hallucination-Free Music: A Reinforcement Learning Preference Optimization Framework for Reliable Song Generationcited by 1
  12. 2025RLAIF-SPA: Structured AI Feedback for Semantic-Prosodic Alignment in Speech Synthesiscited by 1
  13. 2025Iterative Foundation Model Fine-Tuning on Multiple Rewardscited by 0
  14. 2025Aligning Generative Music AI with Human Preferences: Methods and Challengescited by 0
  15. 2025Generative Multi-modal Feedback for Singing Voice Synthesis Evaluationcited by 0
  16. 2026Same Words, Different Judgments: How Preferences Vary Across Modalitiescited by 0
  17. 2026LabelBuddy: An Open Source Music and Audio Language Annotation Tagging Tool Using AI Assistancecited by 0
  18. 2026DDSynth-RL: Audio Synthesizer Inversion via Discrete Diffusion with Reinforcement Learningcited by 0
  19. 2026MuseCritic: Learning Multi-Aspect Song Rewards through Natural-Language Aesthetic Critiquescited by 0
  20. 2026Which Constraints Are Missing? Ask the Verifier: Graded Rewards for Constraint-Following Music Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Improving Text-to-Music Generation with Human Preference Rewards
  2. 2025Auto-Regressive vs Flow-Matching: a Comparative Study of Modeling Paradigms for Text-to-Music Generation
  3. 2023JEN-1: Text-Guided Universal Music Generation with Omnidirectional Diffusion Models
  4. 2026ACE-Step 1.5: Pushing the Boundaries of Open-Source Music Generation
  5. 2026Inferring Value Criteria from Ordinal Preferences: An Iterative In-Context Learning Framework for Music Generation ↗
  6. 2026Instrumental Text-to-Music Generation with Auxiliary Conditioning Branches ↗
  7. 2025JAM: A Tiny Flow-based Song Generator with Fine-grained Controllability and Aesthetic Alignment
  8. 2025DiffRhythm 2: Efficient and High Fidelity Song Generation via Block Flow Matching