# DeepSinger: Singing Voice Synthesis with Data Mined From the Web

arXiv: https://arxiv.org/abs/2007.04590 · HTML: https://mlsanity.com/paper/2007.04590/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Jul 2020.

## Built on by (most cited first, 30 of 32)

- [DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism](https://mlsanity.com/paper/2105.02446/) (2021)
- [MusicAgent: An AI Agent for Music Understanding and Generation with Large Language Models](https://mlsanity.com/paper/2310.11954/) (2023)
- [EMOVIE: A Mandarin Emotion Speech Dataset with a Simple Emotional Text-to-Speech Model](https://mlsanity.com/paper/2106.09317/) (2021)
- [Sinsy: A Deep Neural Network-Based Singing Voice Synthesis System](https://mlsanity.com/paper/2108.02776/) (2021)
- [Deep Performer: Score-to-Audio Music Performance Synthesis](https://mlsanity.com/paper/2202.06034/) (2022)
- [RMSSinger: Realistic-Music-Score based Singing Voice Synthesis](https://mlsanity.com/paper/2305.10686/) (2023)
- [FSD: An Initial Chinese Dataset for Fake Song Detection](https://mlsanity.com/paper/2309.02232/) (2023)
- [FreGrad: Lightweight and Fast Frequency-aware Diffusion Vocoder](https://mlsanity.com/paper/2401.10032/) (2024)
- [Learning the Beauty in Songs: Neural Singing Voice Beautifier](https://mlsanity.com/paper/2202.13277/) (2022)
- [A Melody-Unsupervision Model for Singing Voice Synthesis](https://mlsanity.com/paper/2110.06546/) (2021)
- [WeSinger: Data-augmented Singing Voice Synthesis with Auxiliary Losses](https://mlsanity.com/paper/2203.10750/) (2022)
- [RapVerse: Coherent Vocals and Whole-Body Motions Generations from Text](https://mlsanity.com/paper/2405.20336/) (2024)
- [Translate the Beauty in Songs: Jointly Learning to Align Melody and Translate Lyrics](https://mlsanity.com/paper/2303.15705/) (2023)
- [RefineGAN: Universally Generating Waveform Better than Ground Truth with Highly Accurate Pitch and Intensity Responses](https://mlsanity.com/paper/2111.00962/) (2021)
- [Towards Improving the Expressiveness of Singing Voice Synthesis with BERT Derived Semantic Information](https://mlsanity.com/paper/2308.16836/) (2023)
- [KaraSinger: Score-Free Singing Voice Synthesis with VQ-VAE using Mel-spectrograms](https://mlsanity.com/paper/2110.04005/) (2021)
- [SingAug: Data Augmentation for Singing Voice Synthesis with Cycle-consistent Training Strategy](https://mlsanity.com/paper/2203.17001/) (2022)
- [MakeSinger: A Semi-Supervised Training Method for Data-Efficient Singing Voice Synthesis via Classifier-free Diffusion Guidance](https://mlsanity.com/paper/2406.05965/) (2024)
- [SiFiSinger: A High-Fidelity End-to-End Singing Voice Synthesizer based on Source-filter Model](https://mlsanity.com/paper/2410.12536/) (2024)
- [CoMelSinger: Discrete Token-Based Zero-Shot Singing Synthesis With Structured Melody Control and Guidance](https://mlsanity.com/paper/2509.19883/) (2025)
- [DiTSinger: Scaling Singing Voice Synthesis with Diffusion Transformer and Implicit Alignment](https://mlsanity.com/paper/2510.09016/) (2025)
- [Rapping-Singing Voice Synthesis based on Phoneme-level Prosody Control](https://mlsanity.com/paper/2111.09146/) (2021)
- [BiSinger: Bilingual Singing Voice Synthesis](https://mlsanity.com/paper/2309.14089/) (2023)
- [RDSinger: Reference-based Diffusion Network for Singing Voice Synthesis](https://mlsanity.com/paper/2410.21641/) (2024)
- [SmoothSinger: A Conditional Diffusion Model for Singing Voice Synthesis with Multi-Resolution Architecture](https://mlsanity.com/paper/2506.21478/) (2025)
- [Synchronising speech segments with musical beats in Mandarin and English singing](https://mlsanity.com/paper/2106.10045/) (2021)
- [Elucidate Gender Fairness in Singing Voice Transcription](https://mlsanity.com/paper/2308.02898/) (2023)
- [Enhancing the vocal range of single-speaker singing voice synthesis with melody-unsupervised pre-training](https://mlsanity.com/paper/2309.00284/) (2023)
- [Are Music Foundation Models Better at Singing Voice Deepfake Detection? Far-Better Fuse them with Speech Foundation Models](https://mlsanity.com/paper/2409.14131/) (2024)
- [ConSinger: Efficient High-Fidelity Singing Voice Generation with Minimal Steps](https://mlsanity.com/paper/2410.15342/) (2024)

## Abstract (arXiv)

In this paper, we develop DeepSinger, a multi-lingual multi-singer singing voice synthesis (SVS) system, which is built from scratch using singing training data mined from music websites. The pipeline of DeepSinger consists of several steps, including data crawling, singing and accompaniment separation, lyrics-to-singing alignment, data filtration, and singing modeling. Specifically, we design a lyrics-to-singing alignment model to automatically extract the duration of each phoneme in lyrics starting from coarse-grained sentence level to fine-grained phoneme level, and further design a multi-lingual multi-singer singing model based on a feed-forward Transformer to directly generate linear-spectrograms from lyrics, and synthesize voices using Griffin-Lim. DeepSinger has several advantages over previous SVS systems: 1) to the best of our knowledge, it is the first SVS system that directly mines training data from music websites, 2) the lyrics-to-singing alignment model further avoids any human efforts for alignment labeling and greatly reduces labeling cost, 3) the singing model based on a feed-forward Transformer is simple and efficient, by removing the complicated acoustic feature modeling in parametric synthesis and leveraging a reference encoder to capture the timbre of a singer from noisy singing data, and 4) it can synthesize singing voices in multiple languages and multiple singers. We evaluate DeepSinger on our mined singing dataset that consists of about 92 hours data from 89 singers on three languages (Chinese, Cantonese and English). The results demonstrate that with the singing data purely mined from the Web, DeepSinger can synthesize high-quality singing voices in terms of both pitch accuracy and voice naturalness (footnote: Our audio samples are shown in https://speechresearch.github.io/deepsinger/.)