mlsanityData to 7 Oct 2026

AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining

arXiv 2308.05734 ↗First posted 10 Aug 2023cs.SD cs.AI cs.MMcited by 343

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

Although audio generation shares commonalities across different types of audio, such as speech, music, and sound effects, designing models for each type requires careful consideration of specific objectives and biases that can significantly differ from those of other types. To bring us closer to a unified perspective of audio generation, this paper proposes a framework that utilizes the same learning method for speech, music, and sound effect generation. Our framework introduces a general representation of audio, called "language of audio" (LOA). Any audio can be translated into LOA based on AudioMAE, a self-supervised pre-trained representation learning model. In the generation process, we translate any modalities into LOA by using a GPT-2 model, and we perform self-supervised audio generation learning with a latent diffusion model conditioned on LOA. The proposed framework naturally brings advantages such as in-context learning abilities and reusable self-supervised pretrained AudioMAE and latent diffusion models. Experiments on the major benchmarks of text-to-audio, text-to-music, and text-to-speech demonstrate state-of-the-art or competitive performance against previous approaches. Our code, pretrained model, and demo are available at https://audioldm.github.io/audioldm2.

Key facts. Cited by 343 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 74 of its 99 references that are in the map. First posted on arXiv 10 Aug 2023.

Builds on

74 of 99 references.

Closest ancestors · 11

Same direction, newest first

  1. 2023Make-An-Audio 2: Temporal-Enhanced Text-to-Audio Generation29 May
  2. 2023Text-to-Audio Generation using Instruction-Tuned LLM and Latent Diffusion Model24 Apr
  3. 2023Noise2Music: Text-conditioned Music Generation with Diffusion Models8 Feb
  4. 2023Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models30 Jan
  5. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Models29 Jan
  6. 2022ResGrad: Residual Denoising Diffusion Probabilistic Models for Text to Speech30 Dec
  7. 2022Diffsound: Discrete Diffusion Model for Text-to-sound Generation20 Jul
  8. 2022BinauralGrad: A Two-Stage Conditional Diffusion Probabilistic Model for Binaural Audio Synthesis30 May
  9. 2022InferGrad: Improving Diffusion Models for Vocoder by Considering Inference in Training8 Feb
  10. 2021Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech13 May
  11. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesis21 Sep

Other references · 63

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2020Denoising Diffusion Implicit Modelscited by 9,655
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  10. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  11. 2022Classifier-Free Diffusion Guidancecited by 5,717
  12. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
38 more
  1. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  2. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  3. 2023A Survey of Large Language Modelscited by 2,537
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  5. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  6. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  7. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  8. 2021Image Super-Resolution via Iterative Refinementcited by 1,506
  9. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  10. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  11. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  12. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  13. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
  14. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 707
  15. 2016CNN Architectures for Large-Scale Audio Classificationcited by 701
  16. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
  17. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 620
  18. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  19. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
  20. 2023Simple and Controllable Music Generationcited by 480
  21. 2023MusicLM: Generating Music From Textcited by 442
  22. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 391
  23. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  24. 2022AudioGen: Textually Guided Audio Generationcited by 292
  25. 2022Masked Autoencoders that Listencited by 271
  26. 2020Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Searchcited by 269
  27. 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 266
  28. 2021Efficient Training of Audio Transformers with Patchoutcited by 222
  29. 2023A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPTcited by 221
  30. 2016Professor Forcing: A New Algorithm for Training Recurrent Networkscited by 214
  31. 2016FMA: A Dataset For Music Analysiscited by 205
  32. 2020DDSP: Differentiable Digital Signal Processingcited by 177
  33. 2023MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Trainingcited by 175
  34. 2022NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Qualitycited by 151
  35. 2021Taming Visually Guided Sound Generationcited by 136
  36. 2021BYOL for Audio: Self-Supervised Learning for General-Purpose Audio Representationcited by 106
  37. 2022I Hear Your True Colors: Image Guided Audio Generationcited by 92
  38. 2021VoiceFixer: Toward General Speech Restoration with Neural Vocodercited by 50

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 18; 2024: 109; 2025: 119; 2026: 97.

343 papers.

Sort:

By area:

  1. 2026LTX-2: Efficient Joint Audio-Visual Foundation Modelcited by 213
  2. 2024Stable Audio Opencited by 192
  3. 2024Fast Timing-Conditioned Latent Audio Diffusioncited by 178
  4. 2024MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesiscited by 134
  5. 2024Lumina-Next: Making Lumina-T2X Stronger and Faster with Next-DiTcited by 130
  6. 2023Audiobox: Unified Audio Generation with Natural Language Promptscited by 120
  7. 2024Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Alignerscited by 111
  8. 2024FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Soundscited by 109
  9. 2023Mustango: Toward Controllable Text-to-Music Generationcited by 100
  10. 2024Tango 2: Aligning Diffusion-based Text-to-Audio Generations through Direct Preference Optimizationcited by 97
  11. 2024Long-form music generation with latent diffusioncited by 88
  12. 2025JavisDiT: Joint Audio-Video Diffusion Transformer with Hierarchical Spatio-Temporal Prior Synchronizationcited by 84
38 more
  1. 2025Ovi: Twin Backbone Cross-Modal Fusion for Audio-Video Generationcited by 76
  2. 2024DITTO: Diffusion Inference-Time T-Optimization for Music Generationcited by 73
  3. 2023V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Modelscited by 65
  4. 2023CoDi-2: In-Context, Interleaved, and Interactive Any-to-Any Generationcited by 63
  5. 2024Zero-Shot Unsupervised and Text-Based Audio Editing Using DDPM Inversioncited by 58
  6. 2025UniVerse-1: Unified Audio-Video Generation via Stitching of Expertscited by 58
  7. 2024Auffusion: Leveraging the Power of Diffusion and Large Language Models for Text-to-Audio Generationcited by 54
  8. 2025ACE-Step: A Step Towards Music Generation Foundation Modelcited by 54
  9. 2024Frieren: Efficient Video-to-Audio Generation Network with Rectified Flow Matchingcited by 52
  10. 2024Masked Audio Generation using a Single Non-Autoregressive Transformercited by 50
  11. 2024Video-Guided Foley Sound Generation with Multimodal Controlscited by 49
  12. 2024Read, Watch and Scream! Sound Generation from Text and Videocited by 48
  13. 2024Scaling Transformers for Low-Bitrate High-Quality Speech Codingcited by 48
  14. 2024OmniFlow: Any-to-Any Generation with Multi-Modal Rectified Flowscited by 45
  15. 2024SemantiCodec: An Ultra Low Bitrate Semantic Audio Codec for General Soundcited by 42
  16. 2023AudioSR: Versatile Audio Super-resolution at Scalecited by 39
  17. 2026MOVA: Towards Scalable and Synchronized Video-Audio Generationcited by 39
  18. 2024Align Anything: Training All-Modality Models to Follow Instructions with Language Feedbackcited by 38
  19. 2025Kling-Foley: Multimodal Diffusion Transformer for High-Quality Video-to-Audio Generationcited by 38
  20. 2025MotionGPT3: Human Motion as a Second Modalitycited by 36
  21. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  22. 2024Foundation Models for Music: A Surveycited by 34
  23. 2024Tell What You Hear From What You See -- Video to Audio Generation Through Textcited by 34
  24. 2024Diff-A-Riff: Musical Accompaniment Co-creation via Latent Diffusion Modelscited by 33
  25. 2025Judge Anything: MLLM as a Judge Across Any Modalitycited by 33
  26. 2025ThinkSound: Chain-of-Thought Reasoning in Multimodal Large Language Models for Audio Generation and Editingcited by 33
  27. 2024MusicMagus: Zero-Shot Text-to-Music Editing via Diffusion Modelscited by 32
  28. 2024EzAudio: Enhancing Text-to-Audio Generation with Efficient Diffusion Transformercited by 30
  29. 2023Schrodinger Bridges Beat Diffusion Models on Text-to-Speech Synthesiscited by 29
  30. 2025SongGen: A Single Stage Auto-regressive Transformer for Text-to-Song Generationcited by 29
  31. 2024Taming Data and Transformers for Audio Generationcited by 28
  32. 2024Masked Generative Video-to-Audio Transformers with Enhanced Synchronicitycited by 28
  33. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  34. 2024DITTO-2: Distilled Diffusion Inference-Time T-Optimization for Music Generationcited by 26
  35. 2024Video-to-Audio Generation with Hidden Alignmentcited by 26
  36. 2024Both Ears Wide Open: Towards Language-Driven Spatial Audio Generationcited by 25
  37. 2025AudioX: A Unified Framework for Anything-to-Audio Generationcited by 25
  38. 2023InstructAny2Pix: Flexible Visual Editing via Multimodal Instruction Followingcited by 24

50 most cited of 343 All 343 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026AudioCALM: Continuous Autoregressive Language Modeling for Universal Audio Generation ↗
  2. 2025Audio Generation Through Score-Based Generative Modeling: Design Principles and Implementation
  3. 2023AudioToken: Adaptation of Text-Conditioned Diffusion Models for Audio-to-Image Generation
  4. 2025ControlAudio: Tackling Text-Guided, Timing-Indicated and Intelligible Audio Generation via Progressive Diffusion Modeling
  5. 2025AudioGenie: A Training-Free Multi-Agent Framework for Diverse Multimodality-to-Multiaudio Generation
  6. 2025Audio Palette: A Diffusion Transformer with Multi-Signal Conditioning for Controllable Foley Synthesis
  7. 2023ArchiSound: Audio Generation with Diffusion
  8. 2026From Objectives to Applications: Aligning Architectural Biases in Audio Self-Supervised Learning ↗