# BDDM: Bilateral Denoising Diffusion Models for Fast and High-Quality Speech Synthesis

arXiv: https://arxiv.org/abs/2203.13508 · HTML: https://mlsanity.com/paper/2203.13508/

## Key facts

- Cited by 64 papers in the map (data through 14 Oct 2026).
- Placed in Audio, speech and music, one of 14 directions in Diffusion models.
- Builds on 34 of its 47 references that are in the map.
- First posted on arXiv 25 Mar 2022.

## Placed in

- Diffusion models / [Audio, speech and music](https://mlsanity.com/diffusion/audio-speech-and-music/)

## Builds on (most cited first, 30 of 34)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [DiffWave: A Versatile Diffusion Model for Audio Synthesis](https://mlsanity.com/paper/2009.09761/) (2020)
- [Normalizing Flows for Probabilistic Modeling and Inference](https://mlsanity.com/paper/1912.02762/) (2019)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [Variational Diffusion Models](https://mlsanity.com/paper/2107.00630/) (2021)
- [Improved Techniques for Training Score-Based Generative Models](https://mlsanity.com/paper/2006.09011/) (2020)
- [WaveGrad: Estimating Gradients for Waveform Generation](https://mlsanity.com/paper/2009.00713/) (2020)
- [Sliced Score Matching: A Scalable Approach to Density and Score Estimation](https://mlsanity.com/paper/1905.07088/) (2019)
- [MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis](https://mlsanity.com/paper/1910.06711/) (2019)
- [Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram](https://mlsanity.com/paper/1910.11480/) (2019)
- [WaveGlow: A Flow-based Generative Network for Speech Synthesis](https://mlsanity.com/paper/1811.00002/) (2018)
- [Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design](https://mlsanity.com/paper/1902.00275/) (2019)
- [Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models](https://mlsanity.com/paper/2103.04922/) (2021)
- [Efficient Neural Audio Synthesis](https://mlsanity.com/paper/1802.08435/) (2018)
- [Parallel WaveNet: Fast High-Fidelity Speech Synthesis](https://mlsanity.com/paper/1711.10433/) (2017)
- [On Fast Sampling of Diffusion Probabilistic Models](https://mlsanity.com/paper/2106.00132/) (2021)
- [Learning to Efficiently Sample from Diffusion Probabilistic Models](https://mlsanity.com/paper/2106.03802/) (2021)
- [BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling](https://mlsanity.com/paper/1902.02102/) (2019)

## Built on by (most cited first, 30 of 64)

- [Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Models](https://mlsanity.com/paper/2211.09707/) (2022)
- [PTQD: Accurate Post-Training Quantization for Diffusion Models](https://mlsanity.com/paper/2305.10657/) (2023)
- [ProDiff: Progressive Fast Diffusion Model For High-Quality Text-to-Speech](https://mlsanity.com/paper/2207.06389/) (2022)
- [FastDiff: A Fast Conditional Diffusion Model for High-Quality Speech Synthesis](https://mlsanity.com/paper/2204.09934/) (2022)
- [StoRM: A Diffusion-based Stochastic Regeneration Model for Speech Enhancement and Dereverberation](https://mlsanity.com/paper/2212.11851/) (2022)
- [A Complete Survey on Generative AI (AIGC): Is ChatGPT from GPT-4 to GPT-5 All You Need?](https://mlsanity.com/paper/2303.11717/) (2023)
- [Diffusion Action Segmentation](https://mlsanity.com/paper/2303.17959/) (2023)
- [SA-Solver: Stochastic Adams Solver for Fast Sampling of Diffusion Models](https://mlsanity.com/paper/2309.05019/) (2023)
- [Subspace Diffusion Generative Models](https://mlsanity.com/paper/2205.01490/) (2022)
- [A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AI](https://mlsanity.com/paper/2303.13336/) (2023)
- [Accelerating Diffusion Sampling with Optimized Time Steps](https://mlsanity.com/paper/2402.17376/) (2024)
- [FaceTalk: Audio-Driven Motion Diffusion for Neural Parametric Head Models](https://mlsanity.com/paper/2312.08459/) (2023)
- [DDM2: Self-Supervised Diffusion MRI Denoising with Generative Diffusion Models](https://mlsanity.com/paper/2302.03018/) (2023)
- [Score-based Generative Modeling Secretly Minimizes the Wasserstein Distance](https://mlsanity.com/paper/2212.06359/) (2022)
- [Efficient Neural Music Generation](https://mlsanity.com/paper/2305.15719/) (2023)
- [VoiceFlow: Efficient Text-to-Speech with Rectified Flow Matching](https://mlsanity.com/paper/2309.05027/) (2023)
- [Exact Diffusion Inversion via Bi-directional Integration Approximation](https://mlsanity.com/paper/2307.10829/) (2023)
- [Diffusion Facial Forgery Detection](https://mlsanity.com/paper/2401.15859/) (2024)
- [SpecGrad: Diffusion Probabilistic Model based Neural Vocoder with Adaptive Noise Spectral Shaping](https://mlsanity.com/paper/2203.16749/) (2022)
- [EmoDiff: Intensity Controllable Emotional Text-to-Speech with Soft-Label Guidance](https://mlsanity.com/paper/2211.09496/) (2022)
- [Foundation Models for Music: A Survey](https://mlsanity.com/paper/2408.14340/) (2024)
- [Diffusion Models for Audio Restoration](https://mlsanity.com/paper/2402.09821/) (2024)
- [Generating Visual Scenes from Touch](https://mlsanity.com/paper/2309.15117/) (2023)
- [Conditional Denoising Diffusion for Sequential Recommendation](https://mlsanity.com/paper/2304.11433/) (2023)
- [Distribution Shift Inversion for Out-of-Distribution Prediction](https://mlsanity.com/paper/2306.08328/) (2023)
- [Promptable Game Models: Text-Guided Game Simulation via Masked Diffusion Models](https://mlsanity.com/paper/2303.13472/) (2023)
- [GROOT: Generating Robust Watermark for Diffusion-Model-Based Audio Synthesis](https://mlsanity.com/paper/2407.10471/) (2024)
- [WaveFit: An Iterative and Non-autoregressive Neural Vocoder based on Fixed-Point Iteration](https://mlsanity.com/paper/2210.01029/) (2022)
- [DFADD: The Diffusion and Flow-Matching Based Audio Deepfake Dataset](https://mlsanity.com/paper/2409.08731/) (2024)
- [MeLFusion: Synthesizing Music from Image and Language Cues using Diffusion Models](https://mlsanity.com/paper/2406.04673/) (2024)

## Abstract (arXiv)

Diffusion probabilistic models (DPMs) and their extensions have emerged as competitive generative models yet confront challenges of efficient sampling. We propose a new bilateral denoising diffusion model (BDDM) that parameterizes both the forward and reverse processes with a schedule network and a score network, which can train with a novel bilateral modeling objective. We show that the new surrogate objective can achieve a lower bound of the log marginal likelihood tighter than a conventional surrogate. We also find that BDDM allows inheriting pre-trained score network parameters from any DPMs and consequently enables speedy and stable learning of the schedule network and optimization of a noise schedule for sampling. Our experiments demonstrate that BDDMs can generate high-fidelity audio samples with as few as three sampling steps. Moreover, compared to other state-of-the-art diffusion-based neural vocoders, BDDMs produce comparable or higher quality samples indistinguishable from human speech, notably with only seven sampling steps (143x faster than WaveGrad and 28.6x faster than DiffWave). We release our code at https://github.com/tencent-ailab/bddm.