# Adversarially Trained End-to-end Korean Singing Voice Synthesis System

arXiv: https://arxiv.org/abs/1908.01919 · HTML: https://mlsanity.com/paper/1908.01919/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Aug 2019.

## Built on by (most cited first, 22 of 22)

- [DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism](https://mlsanity.com/paper/2105.02446/) (2021)
- [Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised Representations](https://mlsanity.com/paper/2110.14513/) (2021)
- [GANSpeech: Adversarial Training for High-Fidelity Multi-Speaker Speech Synthesis](https://mlsanity.com/paper/2106.15153/) (2021)
- [Sinsy: A Deep Neural Network-Based Singing Voice Synthesis System](https://mlsanity.com/paper/2108.02776/) (2021)
- [FastPitchFormant: Source-filter based Decomposed Modeling for Speech Synthesis](https://mlsanity.com/paper/2106.15123/) (2021)
- [Learning the Beauty in Songs: Neural Singing Voice Beautifier](https://mlsanity.com/paper/2202.13277/) (2022)
- [HiddenSinger: High-Quality Singing Voice Synthesis via Neural Audio Codec and Latent Diffusion Models](https://mlsanity.com/paper/2306.06814/) (2023)
- [A Melody-Unsupervision Model for Singing Voice Synthesis](https://mlsanity.com/paper/2110.06546/) (2021)
- [SC VALL-E: Style-Controllable Zero-Shot Text to Speech Synthesizer](https://mlsanity.com/paper/2307.10550/) (2023)
- [Mandarin Singing Voice Synthesis with Denoising Diffusion Probabilistic Wasserstein GAN](https://mlsanity.com/paper/2209.10446/) (2022)
- [Singing-Tacotron: Global duration control attention and dynamic filter for End-to-end singing voice synthesis](https://mlsanity.com/paper/2202.07907/) (2022)
- [KaraSinger: Score-Free Singing Voice Synthesis with VQ-VAE using Mel-spectrograms](https://mlsanity.com/paper/2110.04005/) (2021)
- [SingAug: Data Augmentation for Singing Voice Synthesis with Cycle-consistent Training Strategy](https://mlsanity.com/paper/2203.17001/) (2022)
- [MakeSinger: A Semi-Supervised Training Method for Data-Efficient Singing Voice Synthesis via Classifier-free Diffusion Guidance](https://mlsanity.com/paper/2406.05965/) (2024)
- [Rapping-Singing Voice Synthesis based on Phoneme-level Prosody Control](https://mlsanity.com/paper/2111.09146/) (2021)
- [Learning and controlling the source-filter representation of speech with a variational autoencoder](https://mlsanity.com/paper/2204.07075/) (2022)
- [Karaoker: Alignment-free singing voice synthesis with speech training data](https://mlsanity.com/paper/2204.04127/) (2022)
- [Anyone GAN Sing](https://mlsanity.com/paper/2102.11058/) (2021)
- [Pitch Preservation In Singing Voice Synthesis](https://mlsanity.com/paper/2110.05033/) (2021)
- [Singing Voice Synthesis Based on a Musical Note Position-Aware Attention Mechanism](https://mlsanity.com/paper/2212.13703/) (2022)
- [Singing voice synthesis based on frame-level sequence-to-sequence models considering vocal timing deviation](https://mlsanity.com/paper/2301.02262/) (2023)
- [Evaluating Automatic Speech Recognition Systems for Korean Meteorological Experts](https://mlsanity.com/paper/2410.18444/) (2024)

## Abstract (arXiv)

In this paper, we propose an end-to-end Korean singing voice synthesis system from lyrics and a symbolic melody using the following three novel approaches: 1) phonetic enhancement masking, 2) local conditioning of text and pitch to the super-resolution network, and 3) conditional adversarial training. The proposed system consists of two main modules; a mel-synthesis network that generates a mel-spectrogram from the given input information, and a super-resolution network that upsamples the generated mel-spectrogram into a linear-spectrogram. In the mel-synthesis network, phonetic enhancement masking is applied to generate implicit formant masks solely from the input text, which enables a more accurate phonetic control of singing voice. In addition, we show that two other proposed methods -- local conditioning of text and pitch, and conditional adversarial training -- are crucial for a realistic generation of the human singing voice in the super-resolution process. Finally, both quantitative and qualitative evaluations are conducted, confirming the validity of all proposed methods.