mlsanityData to 7 Oct 2026

Prompt-Singer: Controllable Singing-Voice-Synthesis with Natural Language Prompt

arXiv 2403.11780 ↗First posted 18 Mar 2024cs.SD cs.AI cs.LGcited by 12

Not on the map. Why?

Abstract (arXiv)

Recent singing-voice-synthesis (SVS) methods have achieved remarkable audio quality and naturalness, yet they lack the capability to control the style attributes of the synthesized singing explicitly. We propose Prompt-Singer, the first SVS method that enables attribute controlling on singer gender, vocal range and volume with natural language. We adopt a model architecture based on a decoder-only transformer with a multi-scale hierarchy, and design a range-melody decoupled pitch representation that enables text-conditioned vocal range control while keeping melodic accuracy. Furthermore, we explore various experiment settings, including different types of text representations, text encoder fine-tuning, and introducing speech data to alleviate data scarcity, aiming to facilitate further research. Experiments show that our model achieves favorable controlling ability and audio quality. Audio samples are available at http://prompt-singer.github.io .

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 32 of its 39 references that are in the map. First posted on arXiv 18 Mar 2024.

Builds on

32 of 39 references.

References · 32

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  4. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  5. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  6. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  7. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 707
  8. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  9. 2023Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Modelscited by 323
  10. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 264
  11. 2022BigVGAN: A Universal Neural Vocoder with Large-Scale Trainingcited by 245
  12. 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizerscited by 233
20 more
  1. 2023MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformerscited by 164
  2. 2021DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanismcited by 154
  3. 2023UniAudio: An Audio Foundation Model Toward Universal Audio Generationcited by 127
  4. 2020AISHELL-3: A Multi-speaker Mandarin TTS Corpus and the Baselinescited by 111
  5. 2022PromptTTS: Controllable Text-to-Speech with Text Descriptionscited by 106
  6. 2023InstructTTS: Modelling Expressive TTS in Discrete Latent Space with Natural Language Style Promptcited by 90
  7. 2023Make-An-Audio 2: Temporal-Enhanced Text-to-Audio Generationcited by 90
  8. 2021Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpuscited by 60
  9. 2023Natural Language Supervision for General-Purpose Audio Representationscited by 60
  10. 2022Opencpop: A High-Quality Open Source Chinese Popular Song Corpus for Singing Voice Synthesiscited by 53
  11. 2023PromptTTS 2: Describing and Generating Voices with Text Promptcited by 47
  12. 2015THCHS-30 : A Free Chinese Speech Corpuscited by 43
  13. 2020HiFiSinger: Towards High-Fidelity Neural Singing Voice Synthesiscited by 39
  14. 2021VISinger: Variational Inference with Adversarial Learning for End-to-End Singing Voice Synthesiscited by 38
  15. 2020DiDiSpeech: A Large Scale Mandarin Speech Corpuscited by 33
  16. 2021SingGAN: Generative Adversarial Network For High-Fidelity Singing Voice Generationcited by 27
  17. 2023Make-A-Voice: Unified Voice Synthesis With Discrete Representationcited by 27
  18. 2023PromptVC: Flexible Stylistic Voice Conversion in Latent Space Driven by Natural Language Promptscited by 18
  19. 2022WeSinger: Data-augmented Singing Voice Synthesis with Auxiliary Lossescited by 7
  20. 2022WeSinger 2: Fully Parallel Singing Voice Synthesis via Multi-Singer Conditional Adversarial Trainingcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 6; 2026: 1.

12 papers.

  1. 2024TCSinger: Zero-Shot Singing Voice Synthesis with Style Transfer and Multi-Level Style Controlcited by 20
  2. 2024Voice Attribute Editing with Text Promptcited by 9
  3. 2024Accompanied Singing Voice Synthesis with Fully Text-controlled Melodycited by 9
  4. 2025Low-rank Prompt Interaction for Continual Vision-Language Retrievalcited by 8
  5. 2025TCSinger 2: Customizable Multilingual Zero-shot Singing Voice Synthesiscited by 5
  6. 2025Versatile Framework for Song Generation with Prompt-based Controlcited by 4
  7. 2025CoMelSinger: Discrete Token-Based Zero-Shot Singing Synthesis With Structured Melody Control and Guidancecited by 3
  8. 2024LHQ-SVC: Lightweight and High Quality Singing Voice Conversion Modelingcited by 2
  9. 2025Controllable Singing Voice Synthesis using Phoneme-Level Energy Sequencecited by 1
  10. 2024S2Cap: A Benchmark and a Baseline for Singing Style Captioningcited by 0
  11. 2025SLEEPING-DISCO 9M: A large-scale pre-training dataset for generative music modelingcited by 0
  12. 2026UniVoice: A Unified Model for Speech and Singing Voice Generationcited by 0