# REFFLY: Melody-Constrained Lyrics Editing Model

arXiv: https://arxiv.org/abs/2409.00292 · HTML: https://mlsanity.com/paper/2409.00292/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 42 references that are in the map.
- First posted on arXiv 30 Aug 2024.

## Builds on (most cited first, 22 of 22)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Models](https://mlsanity.com/paper/1610.02424/) (2016)
- [Re3: Generating Longer Stories With Recursive Reprompting and Revision](https://mlsanity.com/paper/2210.06774/) (2022)
- [Evaluating Large Language Models on Controlled Generation Tasks](https://mlsanity.com/paper/2310.14542/) (2023)
- [SongMASS: Automatic Song Writing with Pre-training and Alignment Constraint](https://mlsanity.com/paper/2012.05168/) (2020)
- [Character-Centric Story Visualization via Visual Planning and Token Alignment](https://mlsanity.com/paper/2210.08465/) (2022)
- [SongNet: Rigid Formats Controlled Text Generation](https://mlsanity.com/paper/2004.08022/) (2020)
- [DEAM: Dialogue Coherence Evaluation using AMR-based Semantic Manipulations](https://mlsanity.com/paper/2203.09711/) (2022)
- [Zero-shot Sonnet Generation with Discourse-level Planning and Aesthetics Features](https://mlsanity.com/paper/2205.01821/) (2022)
- [HypoGen: Hyperbole Generation with Commonsense and Counterfactual Knowledge](https://mlsanity.com/paper/2109.05097/) (2021)
- [ExPUNations: Augmenting Puns with Keywords and Explanations](https://mlsanity.com/paper/2210.13513/) (2022)
- [Go Back in Time: Generating Flashbacks in Stories with Event Temporal Prompts](https://mlsanity.com/paper/2205.01898/) (2022)
- [ReLyMe: Improving Lyric-to-Melody Generation by Incorporating Lyric-Melody Relationships](https://mlsanity.com/paper/2207.05688/) (2022)
- [Automatic Song Translation for Tonal Languages](https://mlsanity.com/paper/2203.13420/) (2022)
- [Songs Across Borders: Singable and Controllable Neural Lyric Translation](https://mlsanity.com/paper/2305.16816/) (2023)
- [Youling: an AI-Assisted Lyrics Creation System](https://mlsanity.com/paper/2201.06724/) (2022)
- [A Unified Framework for Pun Generation with Humor Principles](https://mlsanity.com/paper/2210.13055/) (2022)
- [A Hierarchical Attention Based Seq2seq Model for Chinese Lyrics Generation](https://mlsanity.com/paper/1906.06481/) (2019)
- [Say What? Collaborative Pop Lyric Generation Using Multitask Transfer Learning](https://mlsanity.com/paper/2111.07592/) (2021)
- [Generate and Revise: Reinforcement Learning in Neural Poetry](https://mlsanity.com/paper/2102.04114/) (2021)
- [SongRewriter: A Chinese Song Rewriting System with Controllable Content and Rhyme Scheme](https://mlsanity.com/paper/2211.15037/) (2022)

## Built on by (most cited first, 2 of 2)

- [EchoFoley: Event-Centric Hierarchical Control for Video Grounded Creative Sound Generation](https://mlsanity.com/paper/2512.24731/) (2025)
- [CoLyricist: Enhancing Lyric Writing with AI through Workflow-Aligned Support](https://mlsanity.com/paper/2602.22606/) (2026)

## Abstract (arXiv)

Automatic melody-to-lyric (M2L) generation aims to create lyrics that align with a given melody. While most previous approaches generate lyrics from scratch, revision, editing plain text draft to fit it into the melody, offers a much more flexible and practical alternative. This enables broad applications, such as generating lyrics from flexible inputs (keywords, themes, or full text that needs refining to be singable), song translation (preserving meaning across languages while keeping the melody intact), or style transfer (adapting lyrics to different genres). This paper introduces REFFLY (REvision Framework For LYrics), the first revision framework for editing and generating melody-aligned lyrics. We train the lyric revision module using our curated synthesized melody-aligned lyrics dataset, enabling it to transform plain text into lyrics that align with a given melody. To further enhance the revision ability, we propose training-free heuristics aimed at preserving both semantic meaning and musical consistency throughout the editing process. Experimental results demonstrate the effectiveness of REFFLY across various tasks (e.g. lyrics generation, song translation), showing that our model outperforms strong baselines, including Lyra (Tian et al., 2023) and GPT-4, by 25% in both musicality and text quality.