# Aligning Language Models for Lyric-to-Melody Generation with Rule-Based Musical Constraints

arXiv: https://arxiv.org/abs/2604.18489 · HTML: https://mlsanity.com/paper/2604.18489/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Preference alignment for recommenders and RAG, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 15 of its 18 references that are in the map.
- First posted on arXiv 20 Apr 2026.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Preference alignment for recommenders and RAG](https://mlsanity.com/alignment/preference-alignment-for-recommenders-and-rag/)

## Builds on (most cited first, 15 of 15)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [Jukebox: A Generative Model for Music](https://mlsanity.com/paper/2005.00341/) (2020)
- [GLM-4-Voice: Towards Intelligent and Human-Like End-to-End Spoken Chatbot](https://mlsanity.com/paper/2412.02612/) (2024)
- [SongMASS: Automatic Song Writing with Pre-training and Alignment Constraint](https://mlsanity.com/paper/2012.05168/) (2020)
- [Seed-Music: A Unified Framework for High Quality and Controlled Music Generation](https://mlsanity.com/paper/2409.09214/) (2024)
- [TeleMelody: Lyric-to-Melody Generation with a Template-Based Two-Stage Method](https://mlsanity.com/paper/2109.09617/) (2021)
- [TechSinger: Technique Controllable Multilingual Singing Voice Synthesis via Flow Matching](https://mlsanity.com/paper/2502.12572/) (2025)
- [ReLyMe: Improving Lyric-to-Melody Generation by Incorporating Lyric-Melody Relationships](https://mlsanity.com/paper/2207.05688/) (2022)
- [Step-Audio-AQAA: a Fully End-to-End Expressive Large Audio Language Model](https://mlsanity.com/paper/2506.08967/) (2025)

## Abstract (arXiv)

Large Language Models (LLMs) show promise in lyric-to-melody generation, but models trained with Supervised Fine-Tuning (SFT) often produce musically implausible melodies with issues like poor rhythm and unsuitable vocal ranges, a phenomenon we term "constraint violation". To address this, we propose a novel alignment framework that instills musical knowledge without human annotation. We define rule-based musical constraints to automatically generate a preference dataset from an SFT model's outputs. The model is then aligned through a sequential process, first using Direct Preference Optimization (DPO) on paired preference data, followed by Kahneman-Tversky Optimization (KTO) on unpaired negative samples. Experimental results demonstrate that our aligned model substantially reduces rule violations and outperforms strong baselines in both objective and subjective evaluations, generating melodies with substantially improved musicality and coherence. An interactive demo with audio comparisons is available at https://arain233.github.io/AligningMelody-demo.