# Hierarchical Diffusion Models for Singing Voice Neural Vocoder

arXiv: https://arxiv.org/abs/2210.07508 · HTML: https://mlsanity.com/paper/2210.07508/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Audio, speech and music, one of 14 directions in Diffusion models.
- Builds on 28 of its 35 references that are in the map.
- First posted on arXiv 14 Oct 2022.

## Placed in

- Diffusion models / [Audio, speech and music](https://mlsanity.com/diffusion/audio-speech-and-music/)

## Builds on (most cited first, 28 of 28)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [DiffWave: A Versatile Diffusion Model for Audio Synthesis](https://mlsanity.com/paper/2009.09761/) (2020)
- [Cascaded Diffusion Models for High Fidelity Image Generation](https://mlsanity.com/paper/2106.15282/) (2021)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [WaveGrad: Estimating Gradients for Waveform Generation](https://mlsanity.com/paper/2009.00713/) (2020)
- [Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram](https://mlsanity.com/paper/1910.11480/) (2019)
- [WaveGlow: A Flow-based Generative Network for Speech Synthesis](https://mlsanity.com/paper/1811.00002/) (2018)
- [Adversarial Audio Synthesis](https://mlsanity.com/paper/1802.04208/) (2018)
- [Efficient Neural Audio Synthesis](https://mlsanity.com/paper/1802.08435/) (2018)
- [AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Loss](https://mlsanity.com/paper/1905.05879/) (2019)
- [Conditional Diffusion Probabilistic Model for Speech Enhancement](https://mlsanity.com/paper/2202.05256/) (2022)
- [SampleRNN: An Unconditional End-to-End Neural Audio Generation Model](https://mlsanity.com/paper/1612.07837/) (2016)
- [PriorGrad: Improving Conditional Denoising Diffusion Models with Data-Dependent Adaptive Prior](https://mlsanity.com/paper/2106.06406/) (2021)
- [Generating High Fidelity Images with Subscale Pixel Networks and Multidimensional Upscaling](https://mlsanity.com/paper/1812.01608/) (2018)
- [StarGANv2-VC: A Diverse, Unsupervised, Non-parallel Framework for Natural-Sounding Voice Conversion](https://mlsanity.com/paper/2107.10394/) (2021)
- [WaveFlow: A Compact Flow-based Model for Raw Audio](https://mlsanity.com/paper/1912.01219/) (2019)
- [VoiceFixer: Toward General Speech Restoration with Neural Vocoder](https://mlsanity.com/paper/2109.13731/) (2021)
- [HiFi-GAN: High-Fidelity Denoising and Dereverberation Based on Speech Deep Features in Adversarial Networks](https://mlsanity.com/paper/2006.05694/) (2020)
- [SpecGrad: Diffusion Probabilistic Model based Neural Vocoder with Adaptive Noise Spectral Shaping](https://mlsanity.com/paper/2203.16749/) (2022)
- [NHSS: A Speech and Singing Parallel Database](https://mlsanity.com/paper/2012.00337/) (2020)
- [Parametric Resynthesis with neural vocoders](https://mlsanity.com/paper/1906.06762/) (2019)
- [Leveraging Symmetrical Convolutional Transformer Networks for Speech to Singing Voice Style Transfer](https://mlsanity.com/paper/2208.12410/) (2022)

## Built on by (most cited first, 10 of 10)

- [From Discrete Tokens to High-Fidelity Audio Using Multi-Band Diffusion](https://mlsanity.com/paper/2308.02560/) (2023)
- [The Ethical Implications of Generative Audio Models: A Systematic Literature Review](https://mlsanity.com/paper/2307.05527/) (2023)
- [FreGrad: Lightweight and Fast Frequency-aware Diffusion Vocoder](https://mlsanity.com/paper/2401.10032/) (2024)
- [Reconstruction of Sound Field through Diffusion Models](https://mlsanity.com/paper/2312.08821/) (2023)
- [Robust One-Shot Singing Voice Conversion](https://mlsanity.com/paper/2210.11096/) (2022)
- [BigVSAN: Enhancing GAN-based Neural Vocoders with Slicing Adversarial Network](https://mlsanity.com/paper/2309.02836/) (2023)
- [PeriodGrad: Towards Pitch-Controllable Neural Vocoder Based on a Diffusion Probabilistic Model](https://mlsanity.com/paper/2402.14692/) (2024)
- [Iteratively Improving Speech Recognition and Voice Conversion](https://mlsanity.com/paper/2305.15055/) (2023)
- [Enhancing Semantic Communication with Deep Generative Models -- An ICASSP Special Session Overview](https://mlsanity.com/paper/2309.02478/) (2023)
- [WaveNeXt 2: ConvNeXt-Based Fast Neural Vocoders With Residual Denoising and Sub-Modeling for GAN and Diffusion Models](https://mlsanity.com/paper/2605.25506/) (2026)

## Abstract (arXiv)

Recent progress in deep generative models has improved the quality of neural vocoders in speech domain. However, generating a high-quality singing voice remains challenging due to a wider variety of musical expressions in pitch, loudness, and pronunciations. In this work, we propose a hierarchical diffusion model for singing voice neural vocoders. The proposed method consists of multiple diffusion models operating in different sampling rates; the model at the lowest sampling rate focuses on generating accurate low-frequency components such as pitch, and other models progressively generate the waveform at higher sampling rates on the basis of the data at the lower sampling rate and acoustic features. Experimental results show that the proposed method produces high-quality singing voices for multiple singers, outperforming state-of-the-art neural vocoders with a similar range of computational costs.