# Sinsy: A Deep Neural Network-Based Singing Voice Synthesis System

arXiv: https://arxiv.org/abs/2108.02776 · HTML: https://mlsanity.com/paper/2108.02776/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 53 references that are in the map.
- First posted on arXiv 5 Aug 2021.

## Builds on (most cited first, 15 of 15)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://mlsanity.com/paper/1904.02882/) (2019)
- [HiFiSinger: Towards High-Fidelity Neural Singing Voice Synthesis](https://mlsanity.com/paper/2009.01776/) (2020)
- [XiaoiceSing: A High-Quality and Integrated Singing Voice Synthesis System](https://mlsanity.com/paper/2006.06261/) (2020)
- [DeepSinger: Singing Voice Synthesis with Data Mined From the Web](https://mlsanity.com/paper/2007.04590/) (2020)
- [Adversarially Trained End-to-end Korean Singing Voice Synthesis System](https://mlsanity.com/paper/1908.01919/) (2019)
- [ByteSing: A Chinese Singing Voice Synthesis System Using Duration Allocated Encoder-Decoder Acoustic Models and WaveRNN Vocoders](https://mlsanity.com/paper/2004.11012/) (2020)
- [Sequence-to-sequence Singing Synthesis Using the Feed-forward Transformer](https://mlsanity.com/paper/1910.09989/) (2019)
- [Singing Synthesis: with a little help from my attention](https://mlsanity.com/paper/1912.05881/) (2019)
- [Singing Voice Synthesis Using Deep Autoregressive Neural Networks for Acoustic Modeling](https://mlsanity.com/paper/1906.08977/) (2019)
- [Fast and High-Quality Singing Voice Synthesis System based on Convolutional Neural Networks](https://mlsanity.com/paper/1910.11690/) (2019)
- [PeriodNet: A non-autoregressive waveform generation model with a structure separating periodic and aperiodic components](https://mlsanity.com/paper/2102.07786/) (2021)
- [Sequence-to-sequence Singing Voice Synthesis with Perceptual Entropy Loss](https://mlsanity.com/paper/2010.12024/) (2020)

## Built on by (most cited first, 14 of 14)

- [The Ethical Implications of Generative Audio Models: A Systematic Literature Review](https://mlsanity.com/paper/2307.05527/) (2023)
- [HiddenSinger: High-Quality Singing Voice Synthesis via Neural Audio Codec and Latent Diffusion Models](https://mlsanity.com/paper/2306.06814/) (2023)
- [WeSinger: Data-augmented Singing Voice Synthesis with Auxiliary Losses](https://mlsanity.com/paper/2203.10750/) (2022)
- [Unsupervised Melody-to-Lyric Generation](https://mlsanity.com/paper/2305.19228/) (2023)
- [JaCappella Corpus: A Japanese a Cappella Vocal Ensemble Corpus](https://mlsanity.com/paper/2211.16028/) (2022)
- [NNSVS: A Neural Network-Based Singing Voice Synthesis Toolkit](https://mlsanity.com/paper/2210.15987/) (2022)
- [PeriodGrad: Towards Pitch-Controllable Neural Vocoder Based on a Diffusion Probabilistic Model](https://mlsanity.com/paper/2402.14692/) (2024)
- [MakeSinger: A Semi-Supervised Training Method for Data-Efficient Singing Voice Synthesis via Classifier-free Diffusion Guidance](https://mlsanity.com/paper/2406.05965/) (2024)
- [SiFiSinger: A High-Fidelity End-to-End Singing Voice Synthesizer based on Source-filter Model](https://mlsanity.com/paper/2410.12536/) (2024)
- [CoMelSinger: Discrete Token-Based Zero-Shot Singing Synthesis With Structured Melody Control and Guidance](https://mlsanity.com/paper/2509.19883/) (2025)
- [Embedding a Differentiable Mel-cepstral Synthesis Filter to a Neural Speech Synthesis System](https://mlsanity.com/paper/2211.11222/) (2022)
- [Singing Voice Synthesis Based on a Musical Note Position-Aware Attention Mechanism](https://mlsanity.com/paper/2212.13703/) (2022)
- [Singing voice synthesis based on frame-level sequence-to-sequence models considering vocal timing deviation](https://mlsanity.com/paper/2301.02262/) (2023)
- [Unsupervised Melody-Guided Lyrics Generation](https://mlsanity.com/paper/2305.07760/) (2023)

## Abstract (arXiv)

This paper presents Sinsy, a deep neural network (DNN)-based singing voice synthesis (SVS) system. In recent years, DNNs have been utilized in statistical parametric SVS systems, and DNN-based SVS systems have demonstrated better performance than conventional hidden Markov model-based ones. SVS systems are required to synthesize a singing voice with pitch and timing that strictly follow a given musical score. Additionally, singing expressions that are not described on the musical score, such as vibrato and timing fluctuations, should be reproduced. The proposed system is composed of four modules: a time-lag model, a duration model, an acoustic model, and a vocoder, and singing voices can be synthesized taking these characteristics of singing voices into account. To better model a singing voice, the proposed system incorporates improved approaches to modeling pitch and vibrato and better training criteria into the acoustic model. In addition, we incorporated PeriodNet, a non-autoregressive neural vocoder with robustness for the pitch, into our systems to generate a high-fidelity singing voice waveform. Moreover, we propose automatic pitch correction techniques for DNN-based SVS to synthesize singing voices with correct pitch even if the training data has out-of-tune phrases. Experimental results show our system can synthesize a singing voice with better timing, more natural vibrato, and correct pitch, and it can achieve better mean opinion scores in subjective evaluation tests.