# A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AI

arXiv: https://arxiv.org/abs/2303.13336 · HTML: https://mlsanity.com/paper/2303.13336/

## Key facts

- Cited by 68 papers in the map (data through 14 Oct 2026).
- Placed in Audio, speech and music, one of 14 directions in Diffusion models.
- Builds on 88 of its 142 references that are in the map.
- First posted on arXiv 23 Mar 2023.

## Placed in

- Diffusion models / [Audio, speech and music](https://mlsanity.com/diffusion/audio-speech-and-music/)

## Builds on (most cited first, 30 of 88)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [DiffWave: A Versatile Diffusion Model for Audio Synthesis](https://mlsanity.com/paper/2009.09761/) (2020)
- [Improved Techniques for Training Score-Based Generative Models](https://mlsanity.com/paper/2006.09011/) (2020)
- [Denoising Diffusion Restoration Models](https://mlsanity.com/paper/2201.11793/) (2022)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2108.02938/) (2021)
- [WaveGrad: Estimating Gradients for Waveform Generation](https://mlsanity.com/paper/2009.00713/) (2020)
- [Tackling the Generative Learning Trilemma with Denoising Diffusion GANs](https://mlsanity.com/paper/2112.07804/) (2021)
- [Tacotron: Towards End-to-End Speech Synthesis](https://mlsanity.com/paper/1703.10135/) (2017)
- [Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech](https://mlsanity.com/paper/2105.06337/) (2021)
- [Cold Diffusion: Inverting Arbitrary Image Transforms Without Noise](https://mlsanity.com/paper/2208.09392/) (2022)
- [Text-to-image Diffusion Models in Generative AI: A Survey](https://mlsanity.com/paper/2303.07909/) (2023)
- [Diffsound: Discrete Diffusion Model for Text-to-sound Generation](https://mlsanity.com/paper/2207.09983/) (2022)
- [Diffusion Probabilistic Modeling for Video Generation](https://mlsanity.com/paper/2203.09481/) (2022)
- [Exploiting Deep Generative Prior for Versatile Image Restoration and Manipulation](https://mlsanity.com/paper/2003.13659/) (2020)
- [Parallel WaveNet: Fast High-Fidelity Speech Synthesis](https://mlsanity.com/paper/1711.10433/) (2017)
- [Speech Enhancement and Dereverberation with Diffusion-based Generative Models](https://mlsanity.com/paper/2208.05830/) (2022)
- [Real Time Speech Enhancement in the Waveform Domain](https://mlsanity.com/paper/2006.12847/) (2020)
- [Image Processing Using Multi-Code GAN Prior](https://mlsanity.com/paper/1912.07116/) (2019)
- [A Survey on Neural Speech Synthesis](https://mlsanity.com/paper/2106.15561/) (2021)
- [SEGAN: Speech Enhancement Generative Adversarial Network](https://mlsanity.com/paper/1703.09452/) (2017)
- [Diff-TTS: A Denoising Diffusion Model for Text-to-Speech](https://mlsanity.com/paper/2104.01409/) (2021)
- [ProDiff: Progressive Fast Diffusion Model For High-Quality Text-to-Speech](https://mlsanity.com/paper/2207.06389/) (2022)

## Built on by (most cited first, 30 of 68)

- [StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models](https://mlsanity.com/paper/2306.07691/) (2023)
- [Why Diffusion Models Don't Memorize: The Role of Implicit Dynamical Regularization in Training](https://mlsanity.com/paper/2505.17638/) (2025)
- [An Overview of Diffusion Models: Applications, Guided Generation, Statistical Rates and Optimization](https://mlsanity.com/paper/2404.07771/) (2024)
- [On the Design Fundamentals of Diffusion Models: A Survey](https://mlsanity.com/paper/2306.04542/) (2023)
- [One Small Step for Generative AI, One Giant Leap for AGI: A Complete Survey on ChatGPT in AIGC Era](https://mlsanity.com/paper/2304.06488/) (2023)
- [Sora as a World Model? A Complete Survey on Text-to-Video Generation](https://mlsanity.com/paper/2403.05131/) (2024)
- [A Survey on Graph Diffusion Models: Generative AI in Science for Molecule, Protein and Material](https://mlsanity.com/paper/2304.01565/) (2023)
- [LLMs Meet Multimodal Generation and Editing: A Survey](https://mlsanity.com/paper/2405.19334/) (2024)
- [Segment Anything Model (SAM) Meets Glass: Mirror and Transparent Objects Cannot Be Easily Detected](https://mlsanity.com/paper/2305.00278/) (2023)
- [Generative AI for Synthetic Data Generation: Methods, Challenges and the Future](https://mlsanity.com/paper/2403.04190/) (2024)
- [Robustness of SAM: Segment Anything Under Corruptions and Beyond](https://mlsanity.com/paper/2306.07713/) (2023)
- [A Survey of Diffusion Based Image Generation Models: Issues and Their Solutions](https://mlsanity.com/paper/2308.13142/) (2023)
- [Training Free Guided Flow Matching with Optimal Control](https://mlsanity.com/paper/2410.18070/) (2024)
- [Training Data Attribution for Diffusion Models](https://mlsanity.com/paper/2306.02174/) (2023)
- [Towards Controllable Speech Synthesis in the Era of Large Language Models: A Systematic Survey](https://mlsanity.com/paper/2412.06602/) (2024)
- [Explainable Generative AI (GenXAI): A Survey, Conceptualization, and Research Agenda](https://mlsanity.com/paper/2404.09554/) (2024)
- [MP-FedCL: Multiprototype Federated Contrastive Learning for Edge Intelligence](https://mlsanity.com/paper/2304.01950/) (2023)
- [DFADD: The Diffusion and Flow-Matching Based Audio Deepfake Dataset](https://mlsanity.com/paper/2409.08731/) (2024)
- [LLaSE-G1: Incentivizing Generalization Capability for LLaMA-based Speech Enhancement](https://mlsanity.com/paper/2503.00493/) (2025)
- [Wasserstein Convergence of Score-based Generative Models under Semiconvexity and Discontinuous Gradients](https://mlsanity.com/paper/2505.03432/) (2025)
- [Robustness of Segment Anything Model (SAM) for Autonomous Driving in Adverse Weather Conditions](https://mlsanity.com/paper/2306.13290/) (2023)
- [Towards Faster Training of Diffusion Models: An Inspiration of A Consistency Phenomenon](https://mlsanity.com/paper/2404.07946/) (2024)
- [Black-box Targeted Adversarial Attack on Segment Anything (SAM)](https://mlsanity.com/paper/2310.10010/) (2023)
- [Smoothed Preference Optimization via ReNoise Inversion for Aligning Diffusion Models with Varied Human Preferences](https://mlsanity.com/paper/2506.02698/) (2025)
- [MPQ-DMv2: Flexible Residual Mixed Precision Quantization for Low-Bit Diffusion Models with Temporal Distillation](https://mlsanity.com/paper/2507.04290/) (2025)
- [Understanding Segment Anything Model: SAM is Biased Towards Texture Rather than Shape](https://mlsanity.com/paper/2311.11465/) (2023)
- [An Overview on Generative AI at Scale with Edge-Cloud Computing](https://mlsanity.com/paper/2306.17170/) (2023)
- [Multi-modal Generative AI: Multi-modal LLMs, Diffusions, and the Unification](https://mlsanity.com/paper/2409.14993/) (2024)
- [Diffuse or Confuse: A Diffusion Deepfake Speech Dataset](https://mlsanity.com/paper/2410.06796/) (2024)
- [CMMD: Contrastive Multi-Modal Diffusion for Video-Audio Conditional Modeling](https://mlsanity.com/paper/2312.05412/) (2023)

## Abstract (arXiv)

Generative AI has demonstrated impressive performance in various fields, among which speech synthesis is an interesting direction. With the diffusion model as the most popular generative model, numerous works have attempted two active tasks: text to speech and speech enhancement. This work conducts a survey on audio diffusion model, which is complementary to existing surveys that either lack the recent progress of diffusion-based speech synthesis or highlight an overall picture of applying diffusion model in multiple fields. Specifically, this work first briefly introduces the background of audio and diffusion model. As for the text-to-speech task, we divide the methods into three categories based on the stage where diffusion model is adopted: acoustic model, vocoder and end-to-end framework. Moreover, we categorize various speech enhancement tasks by either certain signals are removed or added into the input speech. Comparisons of experimental results and discussions are also covered in this survey.