# Joint Learning of Wording and Formatting for Singable Melody-to-Lyric Generation

arXiv: https://arxiv.org/abs/2307.02146 · HTML: https://mlsanity.com/paper/2307.02146/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 75 references that are in the map.
- First posted on arXiv 5 Jul 2023.

## Builds on (most cited first, 26 of 26)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [Deduplicating Training Data Makes Language Models Better](https://mlsanity.com/paper/2107.06499/) (2021)
- [Multilingual Translation with Extensible Multilingual Pretraining and Finetuning](https://mlsanity.com/paper/2008.00401/) (2020)
- [Hybrid Spectrogram and Waveform Source Separation](https://mlsanity.com/paper/2111.03600/) (2021)
- [Lexically Constrained Neural Machine Translation with Levenshtein Transformer](https://mlsanity.com/paper/2004.12681/) (2020)
- [SongMASS: Automatic Song Writing with Pre-training and Alignment Constraint](https://mlsanity.com/paper/2012.05168/) (2020)
- [Controlling the Output Length of Neural Machine Translation](https://mlsanity.com/paper/1910.10408/) (2019)
- [Conditional LSTM-GAN for Melody Generation from Lyrics](https://mlsanity.com/paper/1908.05551/) (2019)
- [SongNet: Rigid Formats Controlled Text Generation](https://mlsanity.com/paper/2004.08022/) (2020)
- [RefSum: Refactoring Neural Summarization](https://mlsanity.com/paper/2104.07210/) (2021)
- [DeepRapper: Neural Rap Generation with Rhyme and Rhythm Modeling](https://mlsanity.com/paper/2107.01875/) (2021)
- [PoeLM: A Meter- and Rhyme-Controllable Language Model for Unsupervised Poetry Generation](https://mlsanity.com/paper/2205.12206/) (2022)
- [Melody-Conditioned Lyrics Generation with SeqGANs](https://mlsanity.com/paper/2010.14709/) (2020)
- [Automatic Song Translation for Tonal Languages](https://mlsanity.com/paper/2203.13420/) (2022)
- [Transfer Learning of wav2vec 2.0 for Automatic Lyric Transcription](https://mlsanity.com/paper/2207.09747/) (2022)
- [Songs Across Borders: Singable and Controllable Neural Lyric Translation](https://mlsanity.com/paper/2305.16816/) (2023)
- [Youling: an AI-Assisted Lyrics Creation System](https://mlsanity.com/paper/2201.06724/) (2022)
- [Integrating Vectorized Lexical Constraints for Neural Machine Translation](https://mlsanity.com/paper/2203.12210/) (2022)
- [Unsupervised Melody-to-Lyric Generation](https://mlsanity.com/paper/2305.19228/) (2023)
- [Natural Language Processing for Music Knowledge Discovery](https://mlsanity.com/paper/1807.02200/) (2018)
- [A Syllable-Structured, Contextually-Based Conditionally Generation of Chinese Lyrics](https://mlsanity.com/paper/1906.09322/) (2019)
- [Input Augmentation Improves Constrained Beam Search for Neural Machine Translation: NTT at WAT 2021](https://mlsanity.com/paper/2106.05450/) (2021)
- [MM-ALT: A Multimodal Automatic Lyric Transcription System](https://mlsanity.com/paper/2207.06127/) (2022)

## Built on by (most cited first, 1 of 1)

- [Foundation Models for Music: A Survey](https://mlsanity.com/paper/2408.14340/) (2024)

## Abstract (arXiv)

Despite progress in melody-to-lyric generation, a substantial singability gap remains between machine-generated lyrics and those written by human lyricists. In this work, we aim to narrow this gap by jointly learning both wording and formatting for melody-to-lyric generation. After general-domain pretraining, our model acquires length awareness through an self-supervised stage trained on a large text-only lyric corpus. During supervised melody-to-lyric training, we introduce multiple auxiliary supervision objective informed by musicological findings on melody--lyric relationships, encouraging the model to capture fine-grained prosodic and structural patterns. Compared with na"ive fine-tuning, our approach improves adherence to line-count and syllable-count requirements by 3.8% and 21.4% absolute, respectively, without degrading text quality. In human evaluation, it achieves 42.2% and 74.2% relative gains in overall quality over two task-specific baselines, underscoring the importance of formatting-aware training for generating singable lyrics.