# SpecGrad: Diffusion Probabilistic Model based Neural Vocoder with Adaptive Noise Spectral Shaping

arXiv: https://arxiv.org/abs/2203.16749 · HTML: https://mlsanity.com/paper/2203.16749/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Placed in Audio, speech and music, one of 14 directions in Diffusion models.
- Builds on 28 of its 39 references that are in the map.
- First posted on arXiv 31 Mar 2022.

## Placed in

- Diffusion models / [Audio, speech and music](https://mlsanity.com/diffusion/audio-speech-and-music/)

## Builds on (most cited first, 28 of 28)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [DiffWave: A Versatile Diffusion Model for Audio Synthesis](https://mlsanity.com/paper/2009.09761/) (2020)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [WaveGrad: Estimating Gradients for Waveform Generation](https://mlsanity.com/paper/2009.00713/) (2020)
- [Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech](https://mlsanity.com/paper/2105.06337/) (2021)
- [MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis](https://mlsanity.com/paper/1910.06711/) (2019)
- [Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram](https://mlsanity.com/paper/1910.11480/) (2019)
- [WaveGlow: A Flow-based Generative Network for Speech Synthesis](https://mlsanity.com/paper/1811.00002/) (2018)
- [Adversarial Audio Synthesis](https://mlsanity.com/paper/1802.04208/) (2018)
- [Efficient Neural Audio Synthesis](https://mlsanity.com/paper/1802.08435/) (2018)
- [It's Raw! Audio Generation with State-Space Models](https://mlsanity.com/paper/2202.09729/) (2022)
- [SampleRNN: An Unconditional End-to-End Neural Audio Generation Model](https://mlsanity.com/paper/1612.07837/) (2016)
- [PriorGrad: Improving Conditional Denoising Diffusion Models with Data-Dependent Adaptive Prior](https://mlsanity.com/paper/2106.06406/) (2021)
- [BDDM: Bilateral Denoising Diffusion Models for Fast and High-Quality Speech Synthesis](https://mlsanity.com/paper/2203.13508/) (2022)
- [Multi-band MelGAN: Faster Waveform Generation for High-Quality Text-to-Speech](https://mlsanity.com/paper/2005.05106/) (2020)
- [WaveFlow: A Compact Flow-based Model for Raw Audio](https://mlsanity.com/paper/1912.01219/) (2019)
- [VoiceFixer: Toward General Speech Restoration with Neural Vocoder](https://mlsanity.com/paper/2109.13731/) (2021)
- [Neural source-filter waveform models for statistical parametric speech synthesis](https://mlsanity.com/paper/1904.12088/) (2019)
- [A Spectral Energy Distance for Parallel Speech Synthesis](https://mlsanity.com/paper/2008.01160/) (2020)
- [HiFi-GAN: High-Fidelity Denoising and Dereverberation Based on Speech Deep Features in Adversarial Networks](https://mlsanity.com/paper/2006.05694/) (2020)
- [InferGrad: Improving Diffusion Models for Vocoder by Considering Inference in Training](https://mlsanity.com/paper/2202.03751/) (2022)
- [A Curated Dataset of Urban Scenes for Audio-Visual Scene Analysis](https://mlsanity.com/paper/2011.00030/) (2020)
- [DF-Conformer: Integrated architecture of Conv-TasNet and Conformer using linear complexity self-attention for speech enhancement](https://mlsanity.com/paper/2106.15813/) (2021)
- [WARP-Q: Quality Prediction For Generative Neural Speech Codecs](https://mlsanity.com/paper/2102.10449/) (2021)
- [Parametric Resynthesis with neural vocoders](https://mlsanity.com/paper/1906.06762/) (2019)
- [Speaker independence of neural vocoders and their effect on parametric resynthesis speech enhancement](https://mlsanity.com/paper/1911.06266/) (2019)

## Built on by (most cited first, 30 of 37)

- [A Survey on Generative Diffusion Model](https://mlsanity.com/paper/2209.02646/) (2022)
- [Speech Enhancement and Dereverberation with Diffusion-based Generative Models](https://mlsanity.com/paper/2208.05830/) (2022)
- [StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models](https://mlsanity.com/paper/2306.07691/) (2023)
- [HouseDiffusion: Vector Floorplan Generation via a Diffusion Model with Discrete and Continuous Denoising](https://mlsanity.com/paper/2211.13287/) (2022)
- [A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AI](https://mlsanity.com/paper/2303.13336/) (2023)
- [StyleDiffusion: Prompt-Embedding Inversion for Text-Based Editing](https://mlsanity.com/paper/2303.15649/) (2023)
- [Generative Pre-training for Speech with Flow Matching](https://mlsanity.com/paper/2310.16338/) (2023)
- [Diffusion-based Generative Speech Source Separation](https://mlsanity.com/paper/2210.17327/) (2022)
- [Imaginary Voice: Face-styled Diffusion Model for Text-to-Speech](https://mlsanity.com/paper/2302.13700/) (2023)
- [Miipher: A Robust Speech Restoration Model Integrating Self-Supervised Speech and Text Representations](https://mlsanity.com/paper/2303.01664/) (2023)
- [ResGrad: Residual Denoising Diffusion Probabilistic Models for Text to Speech](https://mlsanity.com/paper/2212.14518/) (2022)
- [WaveFit: An Iterative and Non-autoregressive Neural Vocoder based on Fixed-Point Iteration](https://mlsanity.com/paper/2210.01029/) (2022)
- [Detecting Out-Of-Distribution Earth Observation Images with Diffusion Models](https://mlsanity.com/paper/2404.12667/) (2024)
- [PeriodWave: Multi-Period Flow Matching for High-Fidelity Waveform Generation](https://mlsanity.com/paper/2408.07547/) (2024)
- [FreGrad: Lightweight and Fast Frequency-aware Diffusion Vocoder](https://mlsanity.com/paper/2401.10032/) (2024)
- [Hierarchical Diffusion Models for Singing Voice Neural Vocoder](https://mlsanity.com/paper/2210.07508/) (2022)
- [DiffSketching: Sketch Control Image Synthesis with Diffusion Models](https://mlsanity.com/paper/2305.18812/) (2023)
- [HierVST: Hierarchical Adaptive Zero-shot Voice Style Transfer](https://mlsanity.com/paper/2307.16171/) (2023)
- [RFWave: Multi-band Rectified Flow for Audio Waveform Reconstruction](https://mlsanity.com/paper/2403.05010/) (2024)
- [Source Separation by Flow Matching](https://mlsanity.com/paper/2505.16119/) (2025)
- [WaveFM: A High-Fidelity and Efficient Vocoder Based on Flow Matching](https://mlsanity.com/paper/2503.16689/) (2025)
- [Robust One-Shot Singing Voice Conversion](https://mlsanity.com/paper/2210.11096/) (2022)
- [BigVSAN: Enhancing GAN-based Neural Vocoders with Slicing Adversarial Network](https://mlsanity.com/paper/2309.02836/) (2023)
- [GLA-Grad: A Griffin-Lim Extended Waveform Generation Diffusion Model](https://mlsanity.com/paper/2402.15516/) (2024)
- [UniWav: Towards Unified Pre-training for Speech Representation Learning and Generation](https://mlsanity.com/paper/2503.00733/) (2025)
- [Accelerating High-Fidelity Waveform Generation via Adversarial Flow Matching Optimization](https://mlsanity.com/paper/2408.08019/) (2024)
- [SF-Speech: Straightened Flow for Zero-Shot Voice Clone](https://mlsanity.com/paper/2410.12399/) (2024)
- [SpecDiff-GAN: A Spectrally-Shaped Noise Diffusion GAN for Speech and Music Synthesis](https://mlsanity.com/paper/2402.01753/) (2024)
- [PeriodGrad: Towards Pitch-Controllable Neural Vocoder Based on a Diffusion Probabilistic Model](https://mlsanity.com/paper/2402.14692/) (2024)
- [Facial Expression-Enhanced TTS: Combining Face Representation and Emotion Intensity for Adaptive Speech](https://mlsanity.com/paper/2409.16203/) (2024)

## Abstract (arXiv)

Neural vocoder using denoising diffusion probabilistic model (DDPM) has been improved by adaptation of the diffusion noise distribution to given acoustic features. In this study, we propose SpecGrad that adapts the diffusion noise so that its time-varying spectral envelope becomes close to the conditioning log-mel spectrogram. This adaptation by time-varying filtering improves the sound quality especially in the high-frequency bands. It is processed in the time-frequency domain to keep the computational cost almost the same as the conventional DDPM-based neural vocoders. Experimental results showed that SpecGrad generates higher-fidelity speech waveform than conventional DDPM-based neural vocoders in both analysis-synthesis and speech enhancement scenarios. Audio demos are available at wavegrad.github.io/specgrad/.