# SongMASS: Automatic Song Writing with Pre-training and Alignment Constraint

arXiv: https://arxiv.org/abs/2012.05168 · HTML: https://mlsanity.com/paper/2012.05168/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Dec 2020.

## Built on by (most cited first, 30 of 41)

- [A Survey of Controllable Text Generation using Transformer-based Pre-trained Language Models](https://mlsanity.com/paper/2201.05337/) (2022)
- [MusicBERT: Symbolic Music Understanding with Large-Scale Pre-Training](https://mlsanity.com/paper/2106.05630/) (2021)
- [Do Language Models Plagiarize?](https://mlsanity.com/paper/2203.07618/) (2022)
- [Museformer: Transformer with Fine- and Coarse-Grained Attention for Music Generation](https://mlsanity.com/paper/2210.10349/) (2022)
- [DiffRhythm: Blazingly Fast and Embarrassingly Simple End-to-End Full-Length Song Generation with Latent Diffusion](https://mlsanity.com/paper/2503.01183/) (2025)
- [DeepRapper: Neural Rap Generation with Rhyme and Rhythm Modeling](https://mlsanity.com/paper/2107.01875/) (2021)
- [TeleMelody: Lyric-to-Melody Generation with a Template-Based Two-Stage Method](https://mlsanity.com/paper/2109.09617/) (2021)
- [MusicAgent: An AI Agent for Music Understanding and Generation with Large Language Models](https://mlsanity.com/paper/2310.11954/) (2023)
- [Loop Copilot: Conducting AI Ensembles for Music Generation and Iterative Editing](https://mlsanity.com/paper/2310.12404/) (2023)
- [PDMX: A Large-Scale Public Domain MusicXML Dataset for Symbolic Music Processing](https://mlsanity.com/paper/2409.10831/) (2024)
- [GETMusic: Generating Any Music Tracks with a Unified Representation and Diffusion Framework](https://mlsanity.com/paper/2305.10841/) (2023)
- [Automatic Song Translation for Tonal Languages](https://mlsanity.com/paper/2203.13420/) (2022)
- [MelodyGLM: Multi-task Pre-training for Symbolic Melody Generation](https://mlsanity.com/paper/2309.10738/) (2023)
- [SongComposer: A Large Language Model for Lyric and Melody Generation in Song Composition](https://mlsanity.com/paper/2402.17645/) (2024)
- [Accompanied Singing Voice Synthesis with Fully Text-controlled Melody](https://mlsanity.com/paper/2407.02049/) (2024)
- [Unsupervised Melody-to-Lyric Generation](https://mlsanity.com/paper/2305.19228/) (2023)
- [PoetryDiffusion: Towards Joint Semantic and Metrical Manipulation in Poetry Generation](https://mlsanity.com/paper/2306.08456/) (2023)
- [Motifs, Phrases, and Beyond: The Modelling of Structure in Symbolic Music Generation](https://mlsanity.com/paper/2403.07995/) (2024)
- [Re-creation of Creations: A New Paradigm for Lyric-to-Melody Generation](https://mlsanity.com/paper/2208.05697/) (2022)
- [Translate the Beauty in Songs: Jointly Learning to Align Melody and Translate Lyrics](https://mlsanity.com/paper/2303.15705/) (2023)
- [Interpreting Song Lyrics with an Audio-Informed Pre-trained Language Model](https://mlsanity.com/paper/2208.11671/) (2022)
- [MeloForm: Generating Melody with Musical Form based on Expert Systems and Neural Networks](https://mlsanity.com/paper/2208.14345/) (2022)
- [A Survey on Music Generation from Single-Modal, Cross-Modal, and Multi-Modal Perspectives](https://mlsanity.com/paper/2504.00837/) (2025)
- [Versatile Framework for Song Generation with Prompt-based Control](https://mlsanity.com/paper/2504.19062/) (2025)
- [SongRewriter: A Chinese Song Rewriting System with Controllable Content and Rhyme Scheme](https://mlsanity.com/paper/2211.15037/) (2022)
- [Lexical Complexity Controlled Sentence Generation](https://mlsanity.com/paper/2211.14540/) (2022)
- [Controllable Lyrics-to-Melody Generation](https://mlsanity.com/paper/2306.02613/) (2023)
- [Syllable-level lyrics generation from melody exploiting character-level language model](https://mlsanity.com/paper/2310.00863/) (2023)
- [MuDiT & MuSiT: Alignment with Colloquial Expression in Description-to-Song Generation](https://mlsanity.com/paper/2407.03188/) (2024)
- [REFFLY: Melody-Constrained Lyrics Editing Model](https://mlsanity.com/paper/2409.00292/) (2024)

## Abstract (arXiv)

Automatic song writing aims to compose a song (lyric and/or melody) by machine, which is an interesting topic in both academia and industry. In automatic song writing, lyric-to-melody generation and melody-to-lyric generation are two important tasks, both of which usually suffer from the following challenges: 1) the paired lyric and melody data are limited, which affects the generation quality of the two tasks, considering a lot of paired training data are needed due to the weak correlation between lyric and melody; 2) Strict alignments are required between lyric and melody, which relies on specific alignment modeling. In this paper, we propose SongMASS to address the above challenges, which leverages masked sequence to sequence (MASS) pre-training and attention based alignment modeling for lyric-to-melody and melody-to-lyric generation. Specifically, 1) we extend the original sentence-level MASS pre-training to song level to better capture long contextual information in music, and use a separate encoder and decoder for each modality (lyric or melody); 2) we leverage sentence-level attention mask and token-level attention constraint during training to enhance the alignment between lyric and melody. During inference, we use a dynamic programming strategy to obtain the alignment between each word/syllable in lyric and note in melody. We pre-train SongMASS on unpaired lyric and melody datasets, and both objective and subjective evaluations demonstrate that SongMASS generates lyric and melody with significantly better quality than the baseline method without pre-training or alignment constraint.