mlsanityData to 7 Oct 2026

A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AI

arXiv 2303.13336 ↗First posted 23 Mar 2023cs.SD cs.AI cs.LGcited by 68

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

Generative AI has demonstrated impressive performance in various fields, among which speech synthesis is an interesting direction. With the diffusion model as the most popular generative model, numerous works have attempted two active tasks: text to speech and speech enhancement. This work conducts a survey on audio diffusion model, which is complementary to existing surveys that either lack the recent progress of diffusion-based speech synthesis or highlight an overall picture of applying diffusion model in multiple fields. Specifically, this work first briefly introduces the background of audio and diffusion model. As for the text-to-speech task, we divide the methods into three categories based on the stage where diffusion model is adopted: acoustic model, vocoder and end-to-end framework. Moreover, we categorize various speech enhancement tasks by either certain signals are removed or added into the input speech. Comparisons of experimental results and discussions are also covered in this survey.

Key facts. Cited by 68 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 88 of its 142 references that are in the map. First posted on arXiv 23 Mar 2023.

Builds on

88 of 142 references.

Closest ancestors · 29

Same direction, newest first

  1. 2022Grad-StyleSpeech: Any-speaker Adaptive Text-to-Speech Synthesis with Diffusion Models17 Nov
  2. 2022Unsupervised vocal dereverberation with diffusion-based generative models8 Nov
  3. 2022Cold Diffusion for Speech Enhancement4 Nov
  4. 2022NoreSpeech: Knowledge Distillation based Conditional Diffusion Model for Noise-robust Expressive TTS4 Nov
  5. 2022Diffusion-based Generative Speech Source Separation31 Oct
  6. 2022Diffiner: A Versatile Diffusion-based Generative Refiner for Speech Enhancement27 Oct
  7. 2022Conditioning and Sampling in Variational Diffusion Models for Speech Super-Resolution27 Oct
  8. 2022Solving Audio Inverse Problems with a Diffusion Model27 Oct
  9. 2022WaveFit: An Iterative and Non-autoregressive Neural Vocoder based on Fixed-Point Iteration3 Oct
  10. 2022Speech Enhancement and Dereverberation with Diffusion-based Generative Models11 Aug
  11. 2022Diffsound: Discrete Diffusion Model for Text-to-sound Generation20 Jul
  12. 2022ProDiff: Progressive Fast Diffusion Model For High-Quality Text-to-Speech13 Jul
17 more
  1. 2022Zero-Shot Voice Conditioning for Denoising Diffusion TTS Models5 Jun
  2. 2022Guided-TTS 2: A Diffusion Model for High-quality Adaptive Text-to-Speech with Untranscribed Data30 May
  3. 2022FastDiff: A Fast Conditional Diffusion Model for High-Quality Speech Synthesis21 Apr
  4. 2022Speech Enhancement with Score-Based Generative Models in the Complex STFT Domain31 Mar
  5. 2022SpecGrad: Diffusion Probabilistic Model based Neural Vocoder with Adaptive Noise Spectral Shaping31 Mar
  6. 2022BDDM: Bilateral Denoising Diffusion Models for Fast and High-Quality Speech Synthesis25 Mar
  7. 2022Conditional Diffusion Probabilistic Model for Speech Enhancement10 Feb
  8. 2022InferGrad: Improving Diffusion Models for Vocoder by Considering Inference in Training8 Feb
  9. 2022DiffGAN-TTS: High-Fidelity and Efficient Text-to-Speech with Denoising Diffusion GANs28 Jan
  10. 2021A Study on Speech Enhancement Based on Diffusion Probabilistic Model25 Jul
  11. 2021CRASH: Raw Audio Score-based Generative Modeling for Controllable High-resolution Drum Sound Synthesis14 Jun
  12. 2021DiffSVC: A Diffusion Probabilistic Model for Singing Voice Conversion28 May
  13. 2021Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech13 May
  14. 2021Restoring degraded speech via a modified diffusion model22 Apr
  15. 2021NU-Wave: A Diffusion Probabilistic Model for Neural Audio Upsampling6 Apr
  16. 2021Diff-TTS: A Denoising Diffusion Model for Text-to-Speech3 Apr
  17. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesis21 Sep

Other references · 59

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2020Denoising Diffusion Implicit Modelscited by 9,655
  3. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  4. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  5. 2017Neural Discrete Representation Learningcited by 4,650
  6. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  7. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  8. 2020Improved Techniques for Training Score-Based Generative Modelscited by 1,036
  9. 2022Denoising Diffusion Restoration Modelscited by 951
  10. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  11. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
  12. 2021ILVR: Conditioning Method for Denoising Diffusion Probabilistic Modelscited by 634
38 more
  1. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 620
  2. 2021Tackling the Generative Learning Trilemma with Denoising Diffusion GANscited by 515
  3. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 487
  4. 2022Cold Diffusion: Inverting Arbitrary Image Transforms Without Noisecited by 275
  5. 2023Text-to-image Diffusion Models in Generative AI: A Surveycited by 264
  6. 2022Diffusion Probabilistic Modeling for Video Generationcited by 237
  7. 2020Exploiting Deep Generative Prior for Versatile Image Restoration and Manipulationcited by 225
  8. 2017Parallel WaveNet: Fast High-Fidelity Speech Synthesiscited by 214
  9. 2019Image Processing Using Multi-Code GAN Priorcited by 185
  10. 2020Real Time Speech Enhancement in the Waveform Domaincited by 183
  11. 2021A Survey on Neural Speech Synthesiscited by 182
  12. 2017SEGAN: Speech Enhancement Generative Adversarial Networkcited by 165
  13. 2021Taming Visually Guided Sound Generationcited by 136
  14. 2019MetricGAN: Generative Adversarial Networks based Black-box Metric Scores Optimization for Speech Enhancementcited by 104
  15. 2019High Fidelity Speech Synthesis with Adversarial Networkscited by 90
  16. 2023A Complete Survey on Generative AI (AIGC): Is ChatGPT from GPT-4 to GPT-5 All You Need?cited by 84
  17. 2017Deep Voice: Real-time Neural Text-to-Speechcited by 83
  18. 2021Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised Representationscited by 81
  19. 2020End-to-End Adversarial Text-to-Speechcited by 77
  20. 2017Deep Voice 2: Multi-Speaker Neural Text-to-Speechcited by 75
  21. 2022Universal Speech Enhancement with Score-based Diffusioncited by 75
  22. 2018ClariNet: Parallel Wave Generation in End-to-End Text-to-Speechcited by 70
  23. 2017Deep Voice 3: Scaling Text-to-Speech with Convolutional Sequence Learningcited by 67
  24. 2020Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modelingcited by 65
  25. 2021WaveGrad 2: Iterative Refinement for Text-to-Speech Synthesiscited by 65
  26. 2021VoiceFixer: Toward General Speech Restoration with Neural Vocodercited by 50
  27. 2019MelNet: A Generative Model for Audio in the Frequency Domaincited by 41
  28. 2019A Recurrent Variational Autoencoder for Speech Enhancementcited by 38
  29. 2020Wave-Tacotron: Spectrogram-free end-to-end text-to-speech synthesiscited by 38
  30. 2022EmoDiff: Intensity Controllable Emotional Text-to-Speech with Soft-Label Guidancecited by 37
  31. 2019Temporal FiLM: Capturing Long-Range Sequence Dependencies with Feature-Wise Modulationscited by 36
  32. 2022NU-Wave 2: A General Neural Audio Upsampling Model for Various Sampling Ratescited by 36
  33. 2022Full-band General Audio Synthesis with Score-based Diffusioncited by 32
  34. 2020Upsampling artifacts in neural audio synthesiscited by 28
  35. 2021Denoising Diffusion Gamma Modelscited by 27
  36. 2017Audio Super Resolution using Neural Networkscited by 26
  37. 2017End-to-End Waveform Utterance Enhancement for Direct Evaluation Metrics Optimization by Fully Convolutional Neural Networkscited by 26
  38. 2021Conditional Sound Generation Using Neural Discrete Time-Frequency Representation Learningcited by 24

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 17; 2024: 21; 2025: 23; 2026: 7.

68 papers.

Sort:

By area:

  1. 2023StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Modelscited by 148
  2. 2025Why Diffusion Models Don't Memorize: The Role of Implicit Dynamical Regularization in Trainingcited by 78
  3. 2024An Overview of Diffusion Models: Applications, Guided Generation, Statistical Rates and Optimizationcited by 72
  4. 2023On the Design Fundamentals of Diffusion Models: A Surveycited by 59
  5. 2023One Small Step for Generative AI, One Giant Leap for AGI: A Complete Survey on ChatGPT in AIGC Eracited by 55
  6. 2024Sora as a World Model? A Complete Survey on Text-to-Video Generationcited by 53
  7. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  8. 2023A Survey on Graph Diffusion Models: Generative AI in Science for Molecule, Protein and Materialcited by 34
  9. 2023Segment Anything Model (SAM) Meets Glass: Mirror and Transparent Objects Cannot Be Easily Detectedcited by 34
  10. 2024Generative AI for Synthetic Data Generation: Methods, Challenges and the Futurecited by 31
  11. 2023Robustness of SAM: Segment Anything Under Corruptions and Beyondcited by 30
  12. 2023Training Data Attribution for Diffusion Modelscited by 24
38 more
  1. 2023A Survey of Diffusion Based Image Generation Models: Issues and Their Solutionscited by 23
  2. 2024Training Free Guided Flow Matching with Optimal Controlcited by 23
  3. 2024Explainable Generative AI (GenXAI): A Survey, Conceptualization, and Research Agendacited by 22
  4. 2024Towards Controllable Speech Synthesis in the Era of Large Language Models: A Systematic Surveycited by 22
  5. 2023MP-FedCL: Multiprototype Federated Contrastive Learning for Edge Intelligencecited by 20
  6. 2024DFADD: The Diffusion and Flow-Matching Based Audio Deepfake Datasetcited by 20
  7. 2025LLaSE-G1: Incentivizing Generalization Capability for LLaMA-based Speech Enhancementcited by 19
  8. 2025Wasserstein Convergence of Score-based Generative Models under Semiconvexity and Discontinuous Gradientscited by 15
  9. 2023Robustness of Segment Anything Model (SAM) for Autonomous Driving in Adverse Weather Conditionscited by 14
  10. 2024Towards Faster Training of Diffusion Models: An Inspiration of A Consistency Phenomenoncited by 14
  11. 2023Black-box Targeted Adversarial Attack on Segment Anything (SAM)cited by 13
  12. 2025Smoothed Preference Optimization via ReNoise Inversion for Aligning Diffusion Models with Varied Human Preferencescited by 13
  13. 2025MPQ-DMv2: Flexible Residual Mixed Precision Quantization for Low-Bit Diffusion Models with Temporal Distillationcited by 12
  14. 2023Understanding Segment Anything Model: SAM is Biased Towards Texture Rather than Shapecited by 10
  15. 2023An Overview on Generative AI at Scale with Edge-Cloud Computingcited by 9
  16. 2024Multi-modal Generative AI: Multi-modal LLMs, Diffusions, and the Unificationcited by 8
  17. 2024Diffuse or Confuse: A Diffusion Deepfake Speech Datasetcited by 8
  18. 2023CMMD: Contrastive Multi-Modal Diffusion for Video-Audio Conditional Modelingcited by 7
  19. 2024Analyzing Neural Network-Based Generative Diffusion Models through Convex Optimizationcited by 7
  20. 2023Attack-SAM: Towards Attacking Segment Anything Model With Adversarial Examplescited by 5
  21. 2024Diffusion Transformer Captures Spatial-Temporal Dependencies: A Theory for Gaussian Process Datacited by 5
  22. 2024Multi-Track MusicLDM: Towards Versatile Music Generation with Latent Diffusion Modelcited by 5
  23. 2025Boost-and-Skip: A Simple Guidance-Free Diffusion for Minority Generationcited by 5
  24. 2025Score Distillation Sampling for Audio: Source Separation, Synthesis, and Beyondcited by 5
  25. 2023SAM Meets UAP: Attacking Segment Anything Model With Universal Adversarial Perturbationcited by 4
  26. 2024An Investigation of Noise Robustness for Flow-Matching-Based Zero-Shot TTScited by 4
  27. 2024TimeAutoDiff: A Unified Framework for Generation, Imputation, Forecasting, and Time-Varying Metadata Conditioning of Heterogeneous Time Series Tabular Datacited by 4
  28. 2025Test-Time Iterative Error Correction for Efficient Diffusion Modelscited by 4
  29. 2024A Literature Review of Literature Reviews in Pattern Analysis and Machine Intelligencecited by 3
  30. 2025Marco-Voice Technical Reportcited by 3
  31. 2025FreeSliders: Training-Free, Modality-Agnostic Concept Sliders for Fine-Grained Diffusion Control in Images, Audio, and Videocited by 3
  32. 2024Diffusion Models for Generating Ballistic Spacecraft Trajectoriescited by 2
  33. 2024Fine-grained length controllable video captioning with ordinal embeddingscited by 2
  34. 2024LatentSpeech: Latent Diffusion for Text-To-Speech Generationcited by 2
  35. 2025SyntheticPop: Attacking Speaker Verification Systems With Synthetic VoicePopscited by 2
  36. 2025Dual Audio-Centric Modality Coupling for Talking Head Generationcited by 2
  37. 2025UmbraTTS: Adapting Text-to-Speech to Environmental Contexts with Flow Matchingcited by 2
  38. 2025A Survey of Pun Generation: Datasets, Evaluations and Methodologiescited by 2

50 most cited of 68 All 68 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Investigating the Design Space of Diffusion Models for Speech Enhancement
  2. 2023Single and Few-step Diffusion for Generative Speech Enhancement
  3. 2022Analysing Diffusion-based Generative Approaches versus Discriminative Approaches for Speech Restoration
  4. 2024Extract and Diffuse: Latent Integration for Improved Diffusion-based Speech and Vocal Enhancement
  5. 2024Diffusion-based Generative Modeling with Discriminative Guidance for Streamable Speech Enhancement
  6. 2025On the Design of Diffusion-based Neural Speech Codecs
  7. 2022StoRM: A Diffusion-based Stochastic Regeneration Model for Speech Enhancement and Dereverberation
  8. 2023AudioToken: Adaptation of Text-Conditioned Diffusion Models for Audio-to-Image Generation