# MPEcho: A Melody and Phoneme-Aware Generative Framework for Controllable Cover Song Generation

arXiv: https://arxiv.org/abs/2607.26698 · HTML: https://mlsanity.com/paper/2607.26698/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 33 references that are in the map.
- First posted on arXiv 29 Jul 2026.

## Builds on (most cited first, 23 of 23)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Qwen2-Audio Technical Report](https://mlsanity.com/paper/2407.10759/) (2024)
- [Jukebox: A Generative Model for Music](https://mlsanity.com/paper/2005.00341/) (2020)
- [DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism](https://mlsanity.com/paper/2105.02446/) (2021)
- [Meta Audiobox Aesthetics: Unified Automatic Quality Assessment for Speech, Music, and Sound](https://mlsanity.com/paper/2502.05139/) (2025)
- [Eliminating Oversaturation and Artifacts of High Guidance Scales in Diffusion Models](https://mlsanity.com/paper/2410.02416/) (2024)
- [ACE-Step: A Step Towards Music Generation Foundation Model](https://mlsanity.com/paper/2506.00045/) (2025)
- [Opencpop: A High-Quality Open Source Chinese Popular Song Corpus for Singing Voice Synthesis](https://mlsanity.com/paper/2201.07429/) (2022)
- [DiffRhythm: Blazingly Fast and Embarrassingly Simple End-to-End Full-Length Song Generation with Latent Diffusion](https://mlsanity.com/paper/2503.01183/) (2025)
- [RMVPE: A Robust Model for Vocal Pitch Estimation in Polyphonic Music](https://mlsanity.com/paper/2306.15412/) (2023)
- [GTSinger: A Global Multi-Technique Singing Corpus with Realistic Music Scores for All Singing Tasks](https://mlsanity.com/paper/2409.13832/) (2024)
- [SongBloom: Coherent Song Generation via Interleaved Autoregressive Sketching and Diffusion Refinement](https://mlsanity.com/paper/2506.07634/) (2025)
- [TechSinger: Technique Controllable Multilingual Singing Voice Synthesis via Flow Matching](https://mlsanity.com/paper/2502.12572/) (2025)
- [Phoneme Segmentation Using Self-Supervised Speech Models](https://mlsanity.com/paper/2211.01461/) (2022)
- [Contrastive Learning-Based Audio to Lyrics Alignment for Multiple Languages](https://mlsanity.com/paper/2306.07744/) (2023)
- [Pop2Piano : Pop Audio-based Piano Cover Generation](https://mlsanity.com/paper/2211.00895/) (2022)
- [MuseControlLite: Multifunctional Music Generation with Lightweight Conditioners](https://mlsanity.com/paper/2506.18729/) (2025)
- [JAM: A Tiny Flow-based Song Generator with Fine-grained Controllability and Aesthetic Alignment](https://mlsanity.com/paper/2507.20880/) (2025)
- [SongPrep: A Preprocessing Framework and End-to-end Model for Full-song Structure Parsing and Lyrics Transcription](https://mlsanity.com/paper/2509.17404/) (2025)
- [Adapting pretrained speech model for Mandarin lyrics transcription and alignment](https://mlsanity.com/paper/2311.12488/) (2023)
- [STARS: A Unified Framework for Singing Transcription, Alignment, and Refined Style Annotation](https://mlsanity.com/paper/2507.06670/) (2025)
- [Etude: Piano Cover Generation with a Three-Stage Approach -- Extract, strucTUralize, and DEcode](https://mlsanity.com/paper/2509.16522/) (2025)

## Abstract (arXiv)

Cover song generation (CSG) should preserve the melodic and linguistic content of a reference song while recreating the remaining musical components. The state-of-the-art model SongEcho utilizes F0 sequences and voiced/unvoiced (V/UV) tags for conditioning; however, implicit linguistic information from V/UV tags cannot guarantee lyric accuracy, leading to a high phoneme error rate (PER). Inspired by singing voice synthesis (SVS), we propose MPEcho, which integrates a phoneme encoder and a length regulator (LR) into the SongEcho framework. By providing explicit phoneme-level conditioning and precise temporal boundaries, MPEcho significantly reduces PER. To enable this, we developed Phonsa, a Whisper-based automatic transcription model that provides high-precision phoneme-level annotations for singing voices, overcoming the scarcity of high-quality audio-phoneme pairs. Experimental results validate the effectiveness of Phonsa for alignment and MPEcho for end-to-end CSG. The audio samples, code and weights can be accessed from https://lonian6.github.io/MPEcho.github.io/.