# Text-to-Audio Generation using Instruction-Tuned LLM and Latent Diffusion Model

arXiv: https://arxiv.org/abs/2304.13731 · HTML: https://mlsanity.com/paper/2304.13731/

## Key facts

- Cited by 145 papers in the map (data through 14 Oct 2026).
- Placed in Audio-language model reasoning, one of 17 directions in LLM reasoning.
- Builds on 32 of its 41 references that are in the map.
- First posted on arXiv 24 Apr 2023.

## Placed in

- LLM reasoning / [Audio-language model reasoning](https://mlsanity.com/lr/audio-language-model-reasoning/)
- Diffusion models / [Audio, speech and music](https://mlsanity.com/diffusion/audio-speech-and-music/)

## Builds on (most cited first, 30 of 32)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [DiffWave: A Versatile Diffusion Model for Audio Synthesis](https://mlsanity.com/paper/2009.09761/) (2020)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [WaveGrad: Estimating Gradients for Waveform Generation](https://mlsanity.com/paper/2009.00713/) (2020)
- [AudioLDM: Text-to-Audio Generation with Latent Diffusion Models](https://mlsanity.com/paper/2301.12503/) (2023)
- [MusicLM: Generating Music From Text](https://mlsanity.com/paper/2301.11325/) (2023)
- [Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech](https://mlsanity.com/paper/2105.06337/) (2021)
- [AudioGen: Textually Guided Audio Generation](https://mlsanity.com/paper/2209.15352/) (2022)
- [WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research](https://mlsanity.com/paper/2303.17395/) (2023)
- [Diffsound: Discrete Diffusion Model for Text-to-sound Generation](https://mlsanity.com/paper/2207.09983/) (2022)
- [Efficient Diffusion Training via Min-SNR Weighting Strategy](https://mlsanity.com/paper/2303.09556/) (2023)
- [Diff-TTS: A Denoising Diffusion Model for Text-to-Speech](https://mlsanity.com/paper/2104.01409/) (2021)
- [FastDiff: A Fast Conditional Diffusion Model for High-Quality Speech Synthesis](https://mlsanity.com/paper/2204.09934/) (2022)
- [Conditional Diffusion Probabilistic Model for Speech Enhancement](https://mlsanity.com/paper/2202.05256/) (2022)
- [Diffusion-Based Voice Conversion with Fast Maximum Likelihood Sampling Scheme](https://mlsanity.com/paper/2109.13821/) (2021)
- [Universal Speech Enhancement with Score-based Diffusion](https://mlsanity.com/paper/2206.03065/) (2022)
- [Learning from Between-class Examples for Deep Sound Recognition](https://mlsanity.com/paper/1711.10282/) (2017)
- [A Study on Speech Enhancement Based on Diffusion Probabilistic Model](https://mlsanity.com/paper/2107.11876/) (2021)

## Built on by (most cited first, 30 of 145)

- [AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining](https://mlsanity.com/paper/2308.05734/) (2023)
- [Fast Timing-Conditioned Latent Audio Diffusion](https://mlsanity.com/paper/2402.04825/) (2024)
- [Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformers](https://mlsanity.com/paper/2405.05945/) (2024)
- [MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis](https://mlsanity.com/paper/2412.15322/) (2024)
- [Lumina-Next: Making Lumina-T2X Stronger and Faster with Next-DiT](https://mlsanity.com/paper/2406.18583/) (2024)
- [LLM4TS: Aligning Pre-Trained LLMs as Data-Efficient Time-Series Forecasters](https://mlsanity.com/paper/2308.08469/) (2023)
- [Audiobox: Unified Audio Generation with Natural Language Prompts](https://mlsanity.com/paper/2312.15821/) (2023)
- [Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Aligners](https://mlsanity.com/paper/2402.17723/) (2024)
- [SyncDiffusion: Coherent Montage via Synchronized Joint Diffusions](https://mlsanity.com/paper/2306.05178/) (2023)
- [FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Sounds](https://mlsanity.com/paper/2407.01494/) (2024)
- [Mustango: Toward Controllable Text-to-Music Generation](https://mlsanity.com/paper/2311.08355/) (2023)
- [A Comprehensive Survey of Hallucination in Large Language, Image, Video and Audio Foundation Models](https://mlsanity.com/paper/2405.09589/) (2024)
- [Tango 2: Aligning Diffusion-based Text-to-Audio Generations through Direct Preference Optimization](https://mlsanity.com/paper/2404.09956/) (2024)
- [Make-An-Audio 2: Temporal-Enhanced Text-to-Audio Generation](https://mlsanity.com/paper/2305.18474/) (2023)
- [TRACE: Temporal Grounding Video LLM via Causal Event Modeling](https://mlsanity.com/paper/2410.05643/) (2024)
- [GPT4Point: A Unified Framework for Point-Language Understanding and Generation](https://mlsanity.com/paper/2312.02980/) (2023)
- [TangoFlux: Super Fast and Faithful Text to Audio Generation with Flow Matching and Clap-Ranked Preference Optimization](https://mlsanity.com/paper/2412.21037/) (2024)
- [LSTPrompt: Large Language Models as Zero-Shot Time Series Forecasters by Long-Short-Term Prompting](https://mlsanity.com/paper/2402.16132/) (2024)
- [Auffusion: Leveraging the Power of Diffusion and Large Language Models for Text-to-Audio Generation](https://mlsanity.com/paper/2401.01044/) (2024)
- [Read, Watch and Scream! Sound Generation from Text and Video](https://mlsanity.com/paper/2407.05551/) (2024)
- [Posterior Distillation Sampling](https://mlsanity.com/paper/2311.13831/) (2023)
- [SemantiCodec: An Ultra Low Bitrate Semantic Audio Codec for General Sound](https://mlsanity.com/paper/2405.00233/) (2024)
- [Separate Anything You Describe](https://mlsanity.com/paper/2308.05037/) (2023)
- [CompA: Addressing the Gap in Compositional Reasoning in Audio-Language Models](https://mlsanity.com/paper/2310.08753/) (2023)
- [Generative Semantic Communication: Diffusion Models Beyond Bit Recovery](https://mlsanity.com/paper/2306.04321/) (2023)
- [JEN-1: Text-Guided Universal Music Generation with Omnidirectional Diffusion Models](https://mlsanity.com/paper/2308.04729/) (2023)
- [AV-DiT: Efficient Audio-Visual Diffusion Transformer for Joint Audio and Video Generation](https://mlsanity.com/paper/2406.07686/) (2024)
- [Kling-Foley: Multimodal Diffusion Transformer for High-Quality Video-to-Audio Generation](https://mlsanity.com/paper/2506.19774/) (2025)
- [LLMs Meet Multimodal Generation and Editing: A Survey](https://mlsanity.com/paper/2405.19334/) (2024)
- [TAVGBench: Benchmarking Text to Audible-Video Generation](https://mlsanity.com/paper/2404.14381/) (2024)

## Abstract (arXiv)

The immense scale of the recent large language models (LLM) allows many interesting properties, such as, instruction- and chain-of-thought-based fine-tuning, that has significantly improved zero- and few-shot performance in many natural language processing (NLP) tasks. Inspired by such successes, we adopt such an instruction-tuned LLM Flan-T5 as the text encoder for text-to-audio (TTA) generation -- a task where the goal is to generate an audio from its textual description. The prior works on TTA either pre-trained a joint text-audio encoder or used a non-instruction-tuned model, such as, T5. Consequently, our latent diffusion model (LDM)-based approach TANGO outperforms the state-of-the-art AudioLDM on most metrics and stays comparable on the rest on AudioCaps test set, despite training the LDM on a 63 times smaller dataset and keeping the text encoder frozen. This improvement might also be attributed to the adoption of audio pressure level-based sound mixing for training set augmentation, whereas the prior methods take a random mix.