mlsanityData to 7 Oct 2026

AudioLDM: Text-to-Audio Generation with Latent Diffusion Models

arXiv 2301.12503 ↗First posted 29 Jan 2023cs.SD cs.AI cs.MMcited by 592

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

Text-to-audio (TTA) system has recently gained attention for its ability to synthesize general audio based on text descriptions. However, previous studies in TTA have limited generation quality with high computational costs. In this study, we propose AudioLDM, a TTA system that is built on a latent space to learn the continuous audio representations from contrastive language-audio pretraining (CLAP) latents. The pretrained CLAP models enable us to train LDMs with audio embedding while providing text embedding as a condition during sampling. By learning the latent representations of audio signals and their compositions without modeling the cross-modal relationship, AudioLDM is advantageous in both generation quality and computational efficiency. Trained on AudioCaps with a single GPU, AudioLDM achieves state-of-the-art TTA performance measured by both objective and subjective metrics (e.g., frechet distance). Moreover, AudioLDM is the first TTA system that enables various text-guided audio manipulations (e.g., style transfer) in a zero-shot fashion. Our implementation and demos are available at https://audioldm.github.io.

Key facts. Cited by 592 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 53 of its 65 references that are in the map. First posted on arXiv 29 Jan 2023.

Builds on

53 of 65 references.

Closest ancestors · 6

Same direction, newest first

  1. 2022ResGrad: Residual Denoising Diffusion Probabilistic Models for Text to Speech30 Dec
  2. 2022Diffsound: Discrete Diffusion Model for Text-to-sound Generation20 Jul
  3. 2022BinauralGrad: A Two-Stage Conditional Diffusion Probabilistic Model for Binaural Audio Synthesis30 May
  4. 2022InferGrad: Improving Diffusion Models for Vocoder by Considering Inference in Training8 Feb
  5. 2021Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech13 May
  6. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesis21 Sep

Other references · 47

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2020Denoising Diffusion Implicit Modelscited by 9,655
  8. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  9. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  10. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  11. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  12. 2022Classifier-Free Diffusion Guidancecited by 5,717
35 more
  1. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  2. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  3. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  4. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  5. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  6. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  7. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  8. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  9. 2021Image Super-Resolution via Iterative Refinementcited by 1,506
  10. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  11. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 707
  12. 2016CNN Architectures for Large-Scale Audio Classificationcited by 701
  13. 2021Score-based Generative Modeling in Latent Spacecited by 634
  14. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 620
  15. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 619
  16. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 563
  17. 2019Clotho: An Audio Captioning Datasetcited by 354
  18. 2022AudioGen: Textually Guided Audio Generationcited by 292
  19. 2020DDSP: Differentiable Digital Signal Processingcited by 177
  20. 2022HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detectioncited by 174
  21. 2022NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Qualitycited by 151
  22. 2021D2C: Diffusion-Denoising Models for Few-shot Conditional Generationcited by 96
  23. 2021PSLA: Improving Audio Tagging with Pretraining, Sampling, Labeling, and Aggregationcited by 81
  24. 2021VoiceFixer: Toward General Speech Restoration with Neural Vocodercited by 50
  25. 2021Decoupling Magnitude and Phase Estimation with Deep ResUNet for Music Source Separationcited by 48
  26. 2022Separate What You Describe: Language-Queried Audio Source Separationcited by 45
  27. 2022Neural Vocoder is All You Need for Speech Super-resolutioncited by 37
  28. 2021Bilateral Denoising Diffusion Modelscited by 36
  29. 2022Full-band General Audio Synthesis with Score-based Diffusioncited by 32
  30. 2017Audio Super Resolution using Neural Networkscited by 26
  31. 2021Conditional Sound Generation Using Neural Discrete Time-Frequency Representation Learningcited by 24
  32. 2021Audio-to-Image Cross-Modal Generationcited by 11
  33. 2022Simple Pooling Front-ends For Efficient Audio Classificationcited by 8
  34. 2022Learning Temporal Resolution in Spectrogram for Audio Classificationcited by 4
  35. 2022Ontology-aware Learning and Evaluation for Audio Taggingcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 107; 2024: 191; 2025: 174; 2026: 120.

592 papers.

Sort:

By area:

  1. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,536
  2. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  3. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 536
  4. 2023Simple and Controllable Music Generationcited by 480
  5. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewardscited by 431
  6. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretrainingcited by 343
  7. 2023Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Modelscited by 323
  8. 2023Your Diffusion Model is Secretly a Zero-Shot Classifiercited by 301
  9. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  10. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  11. 2024Simplifying, Stabilizing and Scaling Continuous-Time Consistency Modelscited by 276
  12. 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 266
38 more
  1. 2026LTX-2: Efficient Joint Audio-Visual Foundation Modelcited by 213
  2. 2023RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulationcited by 206
  3. 2023Any-to-Any Generation via Composable Diffusioncited by 203
  4. 2024Stable Audio Opencited by 192
  5. 2023Mixed-Type Tabular Data Synthesis with Score-based Diffusion in Latent Spacecited by 183
  6. 2024Fast Timing-Conditioned Latent Audio Diffusioncited by 178
  7. 2023Solving Linear Inverse Problems Provably via Posterior Sampling with Latent Diffusion Modelscited by 156
  8. 2023Text-to-Audio Generation using Instruction-Tuned LLM and Latent Diffusion Modelcited by 145
  9. 2024Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformerscited by 137
  10. 2024MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesiscited by 134
  11. 2023Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Modelscited by 133
  12. 2024Lumina-Next: Making Lumina-T2X Stronger and Faster with Next-DiTcited by 130
  13. 2025The Diffusion Dualitycited by 124
  14. 2023Audiobox: Unified Audio Generation with Natural Language Promptscited by 120
  15. 2024Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Alignerscited by 111
  16. 2024FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Soundscited by 109
  17. 2023SyncDiffusion: Coherent Montage via Synchronized Joint Diffusionscited by 108
  18. 2025Inductive Moment Matchingcited by 107
  19. 2024TFG: Unified Training-Free Guidance for Diffusion Modelscited by 103
  20. 2023Mustango: Toward Controllable Text-to-Music Generationcited by 100
  21. 2023BOOT: Data-free Distillation of Denoising Diffusion Models with Bootstrappingcited by 96
  22. 2023MusicLDM: Enhancing Novelty in Text-to-Music Generation Using Beat-Synchronous Mixup Strategiescited by 94
  23. 2023Make-An-Audio 2: Temporal-Enhanced Text-to-Audio Generationcited by 90
  24. 2024Long-form music generation with latent diffusioncited by 88
  25. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 88
  26. 2025Flow Matching Policy Gradientscited by 86
  27. 2025JavisDiT: Joint Audio-Video Diffusion Transformer with Hierarchical Spatio-Temporal Prior Synchronizationcited by 84
  28. 2023AUDIT: Audio Editing by Following Instructions with Latent Diffusion Modelscited by 81
  29. 2023LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPTcited by 77
  30. 2023Spontaneous Symmetry Breaking in Generative Diffusion Modelscited by 74
  31. 2024DITTO: Diffusion Inference-Time T-Optimization for Music Generationcited by 73
  32. 2024Multimodal Alignment and Fusion: A Surveycited by 73
  33. 2023TaleCrafter: Interactive Story Visualization with Multiple Characterscited by 71
  34. 2023Matryoshka Diffusion Modelscited by 70
  35. 2024Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Modelcited by 70
  36. 2023SafeDiffuser: Safe Planning with Diffusion Probabilistic Modelscited by 69
  37. 2023The Song Describer Dataset: a Corpus of Audio Captions for Music-and-Language Evaluationcited by 68
  38. 2024TangoFlux: Super Fast and Faithful Text to Audio Generation with Flow Matching and Clap-Ranked Preference Optimizationcited by 67

50 most cited of 592 All 592 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025ControlAudio: Tackling Text-Guided, Timing-Indicated and Intelligible Audio Generation via Progressive Diffusion Modeling
  2. 2026Fast Text-to-Audio Generation with One-Step Sampling via Energy-Scoring and Auxiliary Contextual Representation Distillation ↗
  3. 2024On the Semantic Latent Space of Diffusion-Based Text-to-Speech Models
  4. 2023AudioToken: Adaptation of Text-Conditioned Diffusion Models for Audio-to-Image Generation
  5. 2026UAT: Unified Audio-Text Diffusion for Audio Generation, Editing, and Captioning
  6. 2024AudioComposer: Towards Fine-grained Audio Generation with Natural Language Descriptions
  7. 2026AudioCALM: Continuous Autoregressive Language Modeling for Universal Audio Generation ↗
  8. 2025From Text to Talk: Audio-Language Model Needs Non-Autoregressive Joint Training