mlsanityData to 7 Oct 2026

Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models

arXiv 2301.12661 ↗First posted 30 Jan 2023cs.SD cs.LG cs.MMcited by 323

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

Large-scale multimodal generative modeling has created milestones in text-to-image and text-to-video generation. Its application to audio still lags behind for two main reasons: the lack of large-scale datasets with high-quality text-audio pairs, and the complexity of modeling long continuous audio data. In this work, we propose Make-An-Audio with a prompt-enhanced diffusion model that addresses these gaps by 1) introducing pseudo prompt enhancement with a distill-then-reprogram approach, it alleviates data scarcity with orders of magnitude concept compositions by using language-free audios; 2) leveraging spectrogram autoencoder to predict the self-supervised audio representation instead of waveforms. Together with robust contrastive language-audio pretraining (CLAP) representations, Make-An-Audio achieves state-of-the-art results in both objective and subjective benchmark evaluation. Moreover, we present its controllability and generalization for X-to-Audio with "No Modality Left Behind", for the first time unlocking the ability to generate high-definition, high-fidelity audios given a user-defined modality input. Audio samples are available at https://Text-to-Audio.github.io

Key facts. Cited by 323 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 53 of its 70 references that are in the map. First posted on arXiv 30 Jan 2023.

Builds on

53 of 70 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Models29 Jan
  2. 2022Diffsound: Discrete Diffusion Model for Text-to-sound Generation20 Jul
  3. 2022ProDiff: Progressive Fast Diffusion Model For High-Quality Text-to-Speech13 Jul
  4. 2022FastDiff: A Fast Conditional Diffusion Model for High-Quality Speech Synthesis21 Apr

Other references · 49

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2020Denoising Diffusion Implicit Modelscited by 9,655
  7. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  10. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  11. 2022Classifier-Free Diffusion Guidancecited by 5,717
  12. 2022Generative Adversarial Networkscited by 5,603
37 more
  1. 2017Neural Discrete Representation Learningcited by 4,650
  2. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  4. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  5. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  6. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  7. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  8. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  9. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  10. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,539
  11. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  12. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  13. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  14. 2021Resolution-robust Large Mask Inpainting with Fourier Convolutionscited by 730
  15. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  16. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
  17. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  18. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  19. 2019Clotho: An Audio Captioning Datasetcited by 354
  20. 2022CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformerscited by 305
  21. 2022AudioGen: Textually Guided Audio Generationcited by 292
  22. 2022Masked Autoencoders that Listencited by 271
  23. 2021Efficient Training of Audio Transformers with Patchoutcited by 222
  24. 2021SSAST: Self-Supervised Audio Spectrogram Transformercited by 166
  25. 2019EdgeConnect: Generative Image Inpainting with Adversarial Edge Learningcited by 161
  26. 2021Taming Visually Guided Sound Generationcited by 136
  27. 2020Rethinking Image Inpainting via a Mutual Encoder-Decoder with Feature Equalizationscited by 102
  28. 2021AdaSpeech: Adaptive Text to Speech for Custom Voicecited by 98
  29. 2020Foley Music: Learning to Generate Music from Videoscited by 82
  30. 2021Audio Retrieval with Natural Language Queries: A Benchmark Studycited by 61
  31. 2020Text-Free Image-to-Speech Synthesis Using Learned Segmental Unitscited by 50
  32. 2020Audeo: Audio Generation for a Silent Performance Videocited by 47
  33. 2022Audio Retrieval with WavText5K and CLAP Trainingcited by 35
  34. 2022TranSpeech: Speech-to-Speech Translation With Bilateral Perturbationcited by 34
  35. 2021What is the ground truth? Reliability of multi-annotator data for audio taggingcited by 29
  36. 2022GenerSpeech: Towards Style Transfer for Generalizable Out-Of-Domain Text-to-Speechcited by 21
  37. 2021Why Do We Click: Visual Impression-aware News Recommendationcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 64; 2024: 130; 2025: 83; 2026: 46.

323 papers.

Sort:

By area:

  1. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  2. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  3. 2023Simple and Controllable Music Generationcited by 480
  4. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 404
  5. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  6. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretrainingcited by 343
  7. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 264
  8. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modelingcited by 213
  9. 2023RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulationcited by 206
  10. 2023Any-to-Any Generation via Composable Diffusioncited by 203
  11. 2024Stable Audio Opencited by 192
  12. 2024Fast Timing-Conditioned Latent Audio Diffusioncited by 178
38 more
  1. 2023ImageBind-LLM: Multi-modality Instruction Tuningcited by 142
  2. 2024Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformerscited by 137
  3. 2024MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesiscited by 134
  4. 2023On Memorization in Diffusion Modelscited by 122
  5. 2023Audiobox: Unified Audio Generation with Natural Language Promptscited by 120
  6. 2024Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Alignerscited by 111
  7. 2024FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Soundscited by 109
  8. 2023SyncDiffusion: Coherent Montage via Synchronized Joint Diffusionscited by 108
  9. 2023VDT: General-purpose Video Diffusion Transformers via Mask Modelingcited by 102
  10. 2024Adaptive Caching for Faster Video Generation with Diffusion Transformerscited by 100
  11. 2024Tango 2: Aligning Diffusion-based Text-to-Audio Generations through Direct Preference Optimizationcited by 97
  12. 2023MusicLDM: Enhancing Novelty in Text-to-Music Generation Using Beat-Synchronous Mixup Strategiescited by 94
  13. 2023Make-An-Audio 2: Temporal-Enhanced Text-to-Audio Generationcited by 90
  14. 2024Long-form music generation with latent diffusioncited by 88
  15. 2024FaithEval: Can Your Language Model Stay Faithful to Context, Even If "The Moon is Made of Marshmallows"cited by 85
  16. 2023Retrieving Multimodal Information for Augmented Generation: A Surveycited by 84
  17. 2025JavisDiT: Joint Audio-Video Diffusion Transformer with Hierarchical Spatio-Temporal Prior Synchronizationcited by 84
  18. 2023AUDIT: Audio Editing by Following Instructions with Latent Diffusion Modelscited by 81
  19. 2023LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPTcited by 77
  20. 2024DITTO: Diffusion Inference-Time T-Optimization for Music Generationcited by 73
  21. 2024Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Modelcited by 70
  22. 2024TangoFlux: Super Fast and Faithful Text to Audio Generation with Flow Matching and Clap-Ranked Preference Optimizationcited by 67
  23. 2023V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Modelscited by 65
  24. 2025A Systematic Survey of Automatic Prompt Optimization Techniquescited by 61
  25. 2024Auffusion: Leveraging the Power of Diffusion and Large Language Models for Text-to-Audio Generationcited by 54
  26. 2024Diffusion Models and Representation Learning: A Surveycited by 53
  27. 2023Efficient Neural Music Generationcited by 50
  28. 2024Masked Audio Generation using a Single Non-Autoregressive Transformercited by 50
  29. 2024ELLA-V: Stable Neural Codec Language Modeling with Alignment-guided Sequence Reorderingcited by 50
  30. 2024Read, Watch and Scream! Sound Generation from Text and Videocited by 48
  31. 2023Posterior Distillation Samplingcited by 45
  32. 2024OmniFlow: Any-to-Any Generation with Multi-Modal Rectified Flowscited by 45
  33. 2023CompA: Addressing the Gap in Compositional Reasoning in Audio-Language Modelscited by 43
  34. 2024SemantiCodec: An Ultra Low Bitrate Semantic Audio Codec for General Soundcited by 42
  35. 2025HunyuanVideo-Foley: Multimodal Diffusion with Representation Alignment for High-Fidelity Foley Audio Generationcited by 42
  36. 2023Generative Semantic Communication: Diffusion Models Beyond Bit Recoverycited by 41
  37. 2023JEN-1: Text-Guided Universal Music Generation with Omnidirectional Diffusion Modelscited by 41
  38. 2024AV-DiT: Efficient Audio-Visual Diffusion Transformer for Joint Audio and Video Generationcited by 39

50 most cited of 323 All 323 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025DreamAudio: Customized Text-to-Audio Generation with Diffusion Models
  2. 2025Audio Generation Through Score-Based Generative Modeling: Design Principles and Implementation
  3. 2023AudioToken: Adaptation of Text-Conditioned Diffusion Models for Audio-to-Image Generation
  4. 2026Diffusion Models for Joint Audio-Video Generation
  5. 2023A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AI
  6. 2026UAT: Unified Audio-Text Diffusion for Audio Generation, Editing, and Captioning
  7. 2026Fast Text-to-Audio Generation with One-Step Sampling via Energy-Scoring and Auxiliary Contextual Representation Distillation ↗
  8. 2026AudioX-Turbo: A Unified Framework for Efficient Anything-to-Audio Generation