mlsanityData to 14 Oct 2026

Text-to-Audio Generation using Instruction-Tuned LLM and Latent Diffusion Model

arXiv 2304.13731 ↗First posted 24 Apr 2023eess.AS cs.AI cs.CLcited by 145

On the map: LLM reasoning → Audio-language model reasoning

also in 1 more
Abstract (arXiv)

The immense scale of the recent large language models (LLM) allows many interesting properties, such as, instruction- and chain-of-thought-based fine-tuning, that has significantly improved zero- and few-shot performance in many natural language processing (NLP) tasks. Inspired by such successes, we adopt such an instruction-tuned LLM Flan-T5 as the text encoder for text-to-audio (TTA) generation -- a task where the goal is to generate an audio from its textual description. The prior works on TTA either pre-trained a joint text-audio encoder or used a non-instruction-tuned model, such as, T5. Consequently, our latent diffusion model (LDM)-based approach TANGO outperforms the state-of-the-art AudioLDM on most metrics and stays comparable on the rest on AudioCaps test set, despite training the LDM on a 63 times smaller dataset and keeping the text encoder frozen. This improvement might also be attributed to the adoption of audio pressure level-based sound mixing for training set augmentation, whereas the prior methods take a random mix.

Key facts. Cited by 145 papers in the map (data through 14 Oct 2026). Placed in Audio-language model reasoning, one of 17 directions in LLM reasoning. Builds on 32 of its 41 references that are in the map. First posted on arXiv 24 Apr 2023.

Builds on

32 of 41 references.

References · 32

  1. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2020Denoising Diffusion Implicit Modelscited by 9,781
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  8. 2022Classifier-Free Diffusion Guidancecited by 5,847
  9. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,527
  10. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  11. 2013Auto-Encoding Variational Bayescited by 4,053
  12. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
20 more
  1. 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
  2. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,354
  3. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  4. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 622
  5. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Modelscited by 599
  6. 2023MusicLM: Generating Music From Textcited by 448
  7. 2021Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speechcited by 447
  8. 2022AudioGen: Textually Guided Audio Generationcited by 295
  9. 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 267
  10. 2022Diffsound: Discrete Diffusion Model for Text-to-sound Generationcited by 257
  11. 2023Efficient Diffusion Training via Min-SNR Weighting Strategycited by 246
  12. 2021Diff-TTS: A Denoising Diffusion Model for Text-to-Speechcited by 157
  13. 2022FastDiff: A Fast Conditional Diffusion Model for High-Quality Speech Synthesiscited by 136
  14. 2022Conditional Diffusion Probabilistic Model for Speech Enhancementcited by 132
  15. 2021Diffusion-Based Voice Conversion with Fast Maximum Likelihood Sampling Schemecited by 100
  16. 2022Universal Speech Enhancement with Score-based Diffusioncited by 75
  17. 2017Learning from Between-class Examples for Deep Sound Recognitioncited by 54
  18. 2021A Study on Speech Enhancement Based on Diffusion Probabilistic Modelcited by 53
  19. 2021Decoupling Magnitude and Phase Estimation with Deep ResUNet for Music Source Separationcited by 48
  20. 2022SRTNet: Time Domain Speech Enhancement Via Stochastic Refinementcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 30; 2024: 67; 2025: 32; 2026: 16.

145 papers.

Sort:

By area:

  1. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretrainingcited by 344
  2. 2024Fast Timing-Conditioned Latent Audio Diffusioncited by 181
  3. 2024Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformerscited by 137
  4. 2024MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesiscited by 134
  5. 2024Lumina-Next: Making Lumina-T2X Stronger and Faster with Next-DiTcited by 132
  6. 2023LLM4TS: Aligning Pre-Trained LLMs as Data-Efficient Time-Series Forecasterscited by 129
  7. 2023Audiobox: Unified Audio Generation with Natural Language Promptscited by 120
  8. 2024Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Alignerscited by 111
  9. 2023SyncDiffusion: Coherent Montage via Synchronized Joint Diffusionscited by 110
  10. 2024FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Soundscited by 110
  11. 2023Mustango: Toward Controllable Text-to-Music Generationcited by 100
  12. 2024A Comprehensive Survey of Hallucination in Large Language, Image, Video and Audio Foundation Modelscited by 100
38 more
  1. 2024Tango 2: Aligning Diffusion-based Text-to-Audio Generations through Direct Preference Optimizationcited by 97
  2. 2023Make-An-Audio 2: Temporal-Enhanced Text-to-Audio Generationcited by 90
  3. 2024TRACE: Temporal Grounding Video LLM via Causal Event Modelingcited by 89
  4. 2023GPT4Point: A Unified Framework for Point-Language Understanding and Generationcited by 71
  5. 2024TangoFlux: Super Fast and Faithful Text to Audio Generation with Flow Matching and Clap-Ranked Preference Optimizationcited by 68
  6. 2024LSTPrompt: Large Language Models as Zero-Shot Time Series Forecasters by Long-Short-Term Promptingcited by 59
  7. 2024Auffusion: Leveraging the Power of Diffusion and Large Language Models for Text-to-Audio Generationcited by 54
  8. 2024Read, Watch and Scream! Sound Generation from Text and Videocited by 48
  9. 2023Posterior Distillation Samplingcited by 46
  10. 2024SemantiCodec: An Ultra Low Bitrate Semantic Audio Codec for General Soundcited by 45
  11. 2023Separate Anything You Describecited by 43
  12. 2023CompA: Addressing the Gap in Compositional Reasoning in Audio-Language Modelscited by 43
  13. 2023Generative Semantic Communication: Diffusion Models Beyond Bit Recoverycited by 41
  14. 2023JEN-1: Text-Guided Universal Music Generation with Omnidirectional Diffusion Modelscited by 41
  15. 2024AV-DiT: Efficient Audio-Visual Diffusion Transformer for Joint Audio and Video Generationcited by 41
  16. 2025Kling-Foley: Multimodal Diffusion Transformer for High-Quality Video-to-Audio Generationcited by 40
  17. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  18. 2024TAVGBench: Benchmarking Text to Audible-Video Generationcited by 34
  19. 2024AccDiffusion: An Accurate Method for Higher-Resolution Image Generationcited by 33
  20. 2024MusicMagus: Zero-Shot Text-to-Music Editing via Diffusion Modelscited by 32
  21. 2024EzAudio: Enhancing Text-to-Audio Generation with Efficient Diffusion Transformercited by 31
  22. 2023WavJourney: Compositional Audio Creation with Large Language Modelscited by 27
  23. 2025AudioX: A Unified Framework for Anything-to-Audio Generationcited by 27
  24. 2024StrokeNUWA: Tokenizing Strokes for Vector Graphic Synthesiscited by 26
  25. 2023Retrieval-Augmented Text-to-Audio Generationcited by 24
  26. 2023VoiceLDM: Text-to-Speech with Environmental Contextcited by 23
  27. 2023ConsistencyTTA: Accelerating Diffusion-Based Text-to-Audio Generation with Consistency Distillationcited by 22
  28. 2023HeadArtist: Text-conditioned 3D Head Generation with Self Score Distillationcited by 22
  29. 2025PatchPilot: A Cost-Efficient Software Engineering Agent with Early Attempts on Formal Verificationcited by 22
  30. 2024AudioLCM: Text-to-Audio Generation with Latent Consistency Modelscited by 21
  31. 2024PMG : Personalized Multimodal Generation with Large Language Modelscited by 19
  32. 2024Language-Driven Interactive Traffic Trajectory Generationcited by 19
  33. 2024MeLFusion: Synthesizing Music from Image and Language Cues using Diffusion Modelscited by 19
  34. 2024Improving Text-To-Audio Models with Synthetic Captionscited by 19
  35. 2024Language-Guided Joint Audio-Visual Editing via One-Shot Adaptationcited by 19
  36. 2024FlashAudio: Rectified Flows for Fast and High-Fidelity Text-to-Audio Generationcited by 19
  37. 2023Audio Generation with Multiple Conditional Diffusion Modelcited by 18
  38. 2024BATON: Aligning Text-to-Audio Model with Human Preference Feedbackcited by 18

50 most cited of 145 All 145 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023AudioToken: Adaptation of Text-Conditioned Diffusion Models for Audio-to-Image Generation
  2. 2024On the Semantic Latent Space of Diffusion-Based Text-to-Speech Models
  3. 2026SwiftAudio: Data-Efficient Caption-Only Distillation for One-Step Text-to-Audio Diffusion-based Generation
  4. 2025ControlAudio: Tackling Text-Guided, Timing-Indicated and Intelligible Audio Generation via Progressive Diffusion Modeling
  5. 2025From Text to Talk: Audio-Language Model Needs Non-Autoregressive Joint Training
  6. 2025Diffusion based Text-to-Music Generation with Global and Local Text based Conditioning
  7. 2025Language Model Based Text-to-Audio Generation: Anti-Causally Aligned Collaborative Residual Transformers
  8. 2024LAFMA: A Latent Flow Matching Model for Text-to-Audio Generation