# Prompt-Singer: Controllable Singing-Voice-Synthesis with Natural Language Prompt

arXiv: https://arxiv.org/abs/2403.11780 · HTML: https://mlsanity.com/paper/2403.11780/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 39 references that are in the map.
- First posted on arXiv 18 Mar 2024.

## Builds on (most cited first, 30 of 32)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [SoundStream: An End-to-End Neural Audio Codec](https://mlsanity.com/paper/2107.03312/) (2021)
- [Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers](https://mlsanity.com/paper/2301.02111/) (2023)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [AudioLM: a Language Modeling Approach to Audio Generation](https://mlsanity.com/paper/2209.03143/) (2022)
- [Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models](https://mlsanity.com/paper/2301.12661/) (2023)
- [AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head](https://mlsanity.com/paper/2304.12995/) (2023)
- [BigVGAN: A Universal Neural Vocoder with Large-Scale Training](https://mlsanity.com/paper/2206.04658/) (2022)
- [NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers](https://mlsanity.com/paper/2304.09116/) (2023)
- [MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers](https://mlsanity.com/paper/2305.07185/) (2023)
- [DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism](https://mlsanity.com/paper/2105.02446/) (2021)
- [UniAudio: An Audio Foundation Model Toward Universal Audio Generation](https://mlsanity.com/paper/2310.00704/) (2023)
- [AISHELL-3: A Multi-speaker Mandarin TTS Corpus and the Baselines](https://mlsanity.com/paper/2010.11567/) (2020)
- [PromptTTS: Controllable Text-to-Speech with Text Descriptions](https://mlsanity.com/paper/2211.12171/) (2022)
- [InstructTTS: Modelling Expressive TTS in Discrete Latent Space with Natural Language Style Prompt](https://mlsanity.com/paper/2301.13662/) (2023)
- [Make-An-Audio 2: Temporal-Enhanced Text-to-Audio Generation](https://mlsanity.com/paper/2305.18474/) (2023)
- [Natural Language Supervision for General-Purpose Audio Representations](https://mlsanity.com/paper/2309.05767/) (2023)
- [Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus](https://mlsanity.com/paper/2112.10358/) (2021)
- [Opencpop: A High-Quality Open Source Chinese Popular Song Corpus for Singing Voice Synthesis](https://mlsanity.com/paper/2201.07429/) (2022)
- [PromptTTS 2: Describing and Generating Voices with Text Prompt](https://mlsanity.com/paper/2309.02285/) (2023)
- [THCHS-30 : A Free Chinese Speech Corpus](https://mlsanity.com/paper/1512.01882/) (2015)
- [HiFiSinger: Towards High-Fidelity Neural Singing Voice Synthesis](https://mlsanity.com/paper/2009.01776/) (2020)
- [VISinger: Variational Inference with Adversarial Learning for End-to-End Singing Voice Synthesis](https://mlsanity.com/paper/2110.08813/) (2021)
- [DiDiSpeech: A Large Scale Mandarin Speech Corpus](https://mlsanity.com/paper/2010.09275/) (2020)
- [SingGAN: Generative Adversarial Network For High-Fidelity Singing Voice Generation](https://mlsanity.com/paper/2110.07468/) (2021)
- [Make-A-Voice: Unified Voice Synthesis With Discrete Representation](https://mlsanity.com/paper/2305.19269/) (2023)
- [PromptVC: Flexible Stylistic Voice Conversion in Latent Space Driven by Natural Language Prompts](https://mlsanity.com/paper/2309.09262/) (2023)

## Built on by (most cited first, 12 of 12)

- [TCSinger: Zero-Shot Singing Voice Synthesis with Style Transfer and Multi-Level Style Control](https://mlsanity.com/paper/2409.15977/) (2024)
- [Voice Attribute Editing with Text Prompt](https://mlsanity.com/paper/2404.08857/) (2024)
- [Accompanied Singing Voice Synthesis with Fully Text-controlled Melody](https://mlsanity.com/paper/2407.02049/) (2024)
- [Low-rank Prompt Interaction for Continual Vision-Language Retrieval](https://mlsanity.com/paper/2501.14369/) (2025)
- [TCSinger 2: Customizable Multilingual Zero-shot Singing Voice Synthesis](https://mlsanity.com/paper/2505.14910/) (2025)
- [Versatile Framework for Song Generation with Prompt-based Control](https://mlsanity.com/paper/2504.19062/) (2025)
- [CoMelSinger: Discrete Token-Based Zero-Shot Singing Synthesis With Structured Melody Control and Guidance](https://mlsanity.com/paper/2509.19883/) (2025)
- [LHQ-SVC: Lightweight and High Quality Singing Voice Conversion Modeling](https://mlsanity.com/paper/2409.08583/) (2024)
- [Controllable Singing Voice Synthesis using Phoneme-Level Energy Sequence](https://mlsanity.com/paper/2509.07038/) (2025)
- [S2Cap: A Benchmark and a Baseline for Singing Style Captioning](https://mlsanity.com/paper/2409.09866/) (2024)
- [SLEEPING-DISCO 9M: A large-scale pre-training dataset for generative music modeling](https://mlsanity.com/paper/2506.14293/) (2025)
- [UniVoice: A Unified Model for Speech and Singing Voice Generation](https://mlsanity.com/paper/2606.05852/) (2026)

## Abstract (arXiv)

Recent singing-voice-synthesis (SVS) methods have achieved remarkable audio quality and naturalness, yet they lack the capability to control the style attributes of the synthesized singing explicitly. We propose Prompt-Singer, the first SVS method that enables attribute controlling on singer gender, vocal range and volume with natural language. We adopt a model architecture based on a decoder-only transformer with a multi-scale hierarchy, and design a range-melody decoupled pitch representation that enables text-conditioned vocal range control while keeping melodic accuracy. Furthermore, we explore various experiment settings, including different types of text representations, text encoder fine-tuning, and introducing speech data to alleviate data scarcity, aiming to facilitate further research. Experiments show that our model achieves favorable controlling ability and audio quality. Audio samples are available at http://prompt-singer.github.io .