mlsanityData to 14 Oct 2026

CoMelSinger: Discrete Token-Based Zero-Shot Singing Synthesis With Structured Melody Control and Guidance

arXiv 2509.19883 ↗First posted 24 Sep 2025cs.SD cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

Singing Voice Synthesis (SVS) aims to generate expressive vocal performances from structured musical inputs such as lyrics and pitch sequences. While recent progress in discrete codec-based speech synthesis has enabled zero-shot generation via in-context learning, directly extending these techniques to SVS remains non-trivial due to the requirement for precise melody control. In particular, prompt-based generation often introduces prosody leakage, where pitch information is inadvertently entangled within the timbre prompt, compromising controllability. We present CoMelSinger, a zero-shot SVS framework that enables structured and disentangled melody control within a discrete codec modeling paradigm. Built on the non-autoregressive MaskGCT architecture, CoMelSinger replaces conventional text inputs with lyric and pitch tokens, preserving in-context generalization while enhancing melody conditioning. To suppress prosody leakage, we propose a coarse-to-fine contrastive learning strategy that explicitly regularizes pitch redundancy between the acoustic prompt and melody input. Furthermore, we incorporate a lightweight encoder-only Singing Voice Transcription (SVT) module to align acoustic tokens with pitch and duration, offering fine-grained frame-level supervision. Experimental results demonstrate that CoMelSinger achieves notable improvements in pitch accuracy, timbre consistency, and zero-shot transferability over competitive baselines. Audio samples are available at https://danny-nus.github.io/CoMelSinger/.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 61 of its 77 references that are in the map. First posted on arXiv 24 Sep 2025.

Builds on

61 of 77 references.

References · 61

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  5. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  6. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  7. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  8. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 649
  9. 2019Symmetric Cross Entropy for Robust Learning with Noisy Labelscited by 500
  10. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
  11. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 334
  12. 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelscited by 288
38 more
  1. 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizerscited by 235
  2. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Modelscited by 226
  3. 2024Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generationcited by 204
  4. 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 167
  5. 2024MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformercited by 161
  6. 2021DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanismcited by 154
  7. 2024VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizerscited by 122
  8. 2023SoundStorm: Efficient Parallel Audio Generationcited by 114
  9. 2020AISHELL-3: A Multi-speaker Mandarin TTS Corpus and the Baselinescited by 111
  10. 2020Unsupervised Speech Decomposition via Triple Information Bottleneckcited by 88
  11. 2021Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpuscited by 60
  12. 2022Opencpop: A High-Quality Open Source Chinese Popular Song Corpus for Singing Voice Synthesiscited by 53
  13. 2023CoMoSpeech: One-Step Speech and Singing Voice Synthesis via Consistency Modelcited by 45
  14. 2021VISinger: Variational Inference with Adversarial Learning for End-to-End Singing Voice Synthesiscited by 38
  15. 2024VALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via Monotonic Alignmentcited by 38
  16. 2020XiaoiceSing: A High-Quality and Integrated Singing Voice Synthesis Systemcited by 33
  17. 2020DeepSinger: Singing Voice Synthesis with Data Mined From the Webcited by 32
  18. 2022Robust Disentangled Variational Speech Representation Learning for Zero-shot Voice Conversioncited by 28
  19. 2021SingGAN: Generative Adversarial Network For High-Fidelity Singing Voice Generationcited by 27
  20. 2023Make-A-Voice: Unified Voice Synthesis With Discrete Representationcited by 27
  21. 2022AVQVC: One-shot Voice Conversion by Vector Quantization with applying contrastive learningcited by 24
  22. 2023StyleSinger: Style Transfer for Out-of-Domain Singing Voice Synthesiscited by 21
  23. 2025TechSinger: Technique Controllable Multilingual Singing Voice Synthesis via Flow Matchingcited by 19
  24. 2023CLAPSpeech: Learning Prosody from Text Context with Contrastive Language-Audio Pre-trainingcited by 18
  25. 2021Sinsy: A Deep Neural Network-Based Singing Voice Synthesis Systemcited by 14
  26. 2022VISinger 2: High-Fidelity End-to-End Singing Voice Synthesis Enhanced by Digital Signal Processing Synthesizercited by 14
  27. 2019WGANSing: A Multi-Voice Singing Voice Synthesizer Based on the Wasserstein-GANcited by 13
  28. 2023RMSSinger: Realistic-Music-Score based Singing Voice Synthesiscited by 12
  29. 2024Prompt-Singer: Controllable Singing-Voice-Synthesis with Natural Language Promptcited by 12
  30. 2022Disentangleing Content and Fine-grained Prosody Information via Hybrid ASR Bottleneck Features for Voice Conversioncited by 10
  31. 2024Robust Singing Voice Transcription Serves Synthesiscited by 10
  32. 2024Accelerating Codec-based Speech Synthesis with Multi-Token Prediction and Speculative Decodingcited by 10
  33. 2021Hierarchical prosody modeling and control in non-autoregressive parallel neural TTScited by 9
  34. 2023HiddenSinger: High-Quality Singing Voice Synthesis via Neural Audio Codec and Latent Diffusion Modelscited by 9
  35. 2023Learning Speech Representation From Contrastive Token-Acoustic Pretrainingcited by 9
  36. 2023DCTTS: Discrete Diffusion Model with Contrastive Learning for Text-to-speech Generationcited by 9
  37. 2024SingMOS: An extensive Open-Source Singing Voice Dataset for MOS Predictioncited by 9
  38. 2021Learning De-identified Representations of Prosody from Raw Audiocited by 8

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.