CoMelSinger: Discrete Token-Based Zero-Shot Singing Synthesis With Structured Melody Control and Guidance
Not on the map. Why?
Abstract (arXiv)
Singing Voice Synthesis (SVS) aims to generate expressive vocal performances from structured musical inputs such as lyrics and pitch sequences. While recent progress in discrete codec-based speech synthesis has enabled zero-shot generation via in-context learning, directly extending these techniques to SVS remains non-trivial due to the requirement for precise melody control. In particular, prompt-based generation often introduces prosody leakage, where pitch information is inadvertently entangled within the timbre prompt, compromising controllability. We present CoMelSinger, a zero-shot SVS framework that enables structured and disentangled melody control within a discrete codec modeling paradigm. Built on the non-autoregressive MaskGCT architecture, CoMelSinger replaces conventional text inputs with lyric and pitch tokens, preserving in-context generalization while enhancing melody conditioning. To suppress prosody leakage, we propose a coarse-to-fine contrastive learning strategy that explicitly regularizes pitch redundancy between the acoustic prompt and melody input. Furthermore, we incorporate a lightweight encoder-only Singing Voice Transcription (SVT) module to align acoustic tokens with pitch and duration, offering fine-grained frame-level supervision. Experimental results demonstrate that CoMelSinger achieves notable improvements in pitch accuracy, timbre consistency, and zero-shot transferability over competitive baselines. Audio samples are available at https://danny-nus.github.io/CoMelSinger/.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 61 of its 77 references that are in the map. First posted on arXiv 24 Sep 2025.
Builds on
61 of 77 references.
References · 61
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
- 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
- 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
- 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
- 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
- 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
- 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 649
- 2019Symmetric Cross Entropy for Robust Learning with Noisy Labelscited by 500
- 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
- 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 334
- 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelscited by 288
38 more
- 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizerscited by 235
- 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Modelscited by 226
- 2024Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generationcited by 204
- 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 167
- 2024MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformercited by 161
- 2021DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanismcited by 154
- 2024VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizerscited by 122
- 2023SoundStorm: Efficient Parallel Audio Generationcited by 114
- 2020AISHELL-3: A Multi-speaker Mandarin TTS Corpus and the Baselinescited by 111
- 2020Unsupervised Speech Decomposition via Triple Information Bottleneckcited by 88
- 2021Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpuscited by 60
- 2022Opencpop: A High-Quality Open Source Chinese Popular Song Corpus for Singing Voice Synthesiscited by 53
- 2023CoMoSpeech: One-Step Speech and Singing Voice Synthesis via Consistency Modelcited by 45
- 2021VISinger: Variational Inference with Adversarial Learning for End-to-End Singing Voice Synthesiscited by 38
- 2024VALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via Monotonic Alignmentcited by 38
- 2020XiaoiceSing: A High-Quality and Integrated Singing Voice Synthesis Systemcited by 33
- 2020DeepSinger: Singing Voice Synthesis with Data Mined From the Webcited by 32
- 2022Robust Disentangled Variational Speech Representation Learning for Zero-shot Voice Conversioncited by 28
- 2021SingGAN: Generative Adversarial Network For High-Fidelity Singing Voice Generationcited by 27
- 2023Make-A-Voice: Unified Voice Synthesis With Discrete Representationcited by 27
- 2022AVQVC: One-shot Voice Conversion by Vector Quantization with applying contrastive learningcited by 24
- 2023StyleSinger: Style Transfer for Out-of-Domain Singing Voice Synthesiscited by 21
- 2025TechSinger: Technique Controllable Multilingual Singing Voice Synthesis via Flow Matchingcited by 19
- 2023CLAPSpeech: Learning Prosody from Text Context with Contrastive Language-Audio Pre-trainingcited by 18
- 2021Sinsy: A Deep Neural Network-Based Singing Voice Synthesis Systemcited by 14
- 2022VISinger 2: High-Fidelity End-to-End Singing Voice Synthesis Enhanced by Digital Signal Processing Synthesizercited by 14
- 2019WGANSing: A Multi-Voice Singing Voice Synthesizer Based on the Wasserstein-GANcited by 13
- 2023RMSSinger: Realistic-Music-Score based Singing Voice Synthesiscited by 12
- 2024Prompt-Singer: Controllable Singing-Voice-Synthesis with Natural Language Promptcited by 12
- 2022Disentangleing Content and Fine-grained Prosody Information via Hybrid ASR Bottleneck Features for Voice Conversioncited by 10
- 2024Robust Singing Voice Transcription Serves Synthesiscited by 10
- 2024Accelerating Codec-based Speech Synthesis with Multi-Token Prediction and Speculative Decodingcited by 10
- 2021Hierarchical prosody modeling and control in non-autoregressive parallel neural TTScited by 9
- 2023HiddenSinger: High-Quality Singing Voice Synthesis via Neural Audio Codec and Latent Diffusion Modelscited by 9
- 2023Learning Speech Representation From Contrastive Token-Acoustic Pretrainingcited by 9
- 2023DCTTS: Discrete Diffusion Model with Contrastive Learning for Text-to-speech Generationcited by 9
- 2024SingMOS: An extensive Open-Source Singing Voice Dataset for MOS Predictioncited by 9
- 2021Learning De-identified Representations of Prosody from Raw Audiocited by 8
50 most cited of 61 All 61 (JSON)
Built on by
Citing papers per year: 2025: 1; 2026: 2.
3 papers.
- 2025YingMusic-Singer: Zero-shot Singing Voice Synthesis and Editing with Annotation-free Melody Guidancecited by 5
- 2026Remix the Timbre: Diffusion-Based Style Transfer Across Polyphonic Stemscited by 0
- 2026Vibrato Expression Control for Singing Voice Conversion with Improving Independent Controlcited by 0
- 2026Vibrato Expression Control for Singing Voice Conversion with Improving Independent Control15 Jun 2026
- 2026Remix the Timbre: Diffusion-Based Style Transfer Across Polyphonic Stems10 May 2026
- 2025YingMusic-Singer: Zero-shot Singing Voice Synthesis and Editing with Annotation-free Melody Guidance4 Dec 2025