mlsanityData to 7 Oct 2026

Retrieval-Augmented Text-to-Audio Generation

arXiv 2309.08051 ↗First posted 14 Sep 2023cs.SD cs.AI cs.MMcited by 23

On the map: Retrieval-augmented generation → Conversational and speech RAG

Abstract (arXiv)

Despite recent progress in text-to-audio (TTA) generation, we show that the state-of-the-art models, such as AudioLDM, trained on datasets with an imbalanced class distribution, such as AudioCaps, are biased in their generation performance. Specifically, they excel in generating common audio classes while underperforming in the rare ones, thus degrading the overall generation performance. We refer to this problem as long-tailed text-to-audio generation. To address this issue, we propose a simple retrieval-augmented approach for TTA models. Specifically, given an input text prompt, we first leverage a Contrastive Language Audio Pretraining (CLAP) model to retrieve relevant text-audio pairs. The features of the retrieved audio-text data are then used as additional conditions to guide the learning of TTA models. We enhance AudioLDM with our proposed approach and denote the resulting augmented system as Re-AudioLDM. On the AudioCaps dataset, Re-AudioLDM achieves a state-of-the-art Frechet Audio Distance (FAD) of 1.37, outperforming the existing approaches by a large margin. Furthermore, we show that Re-AudioLDM can generate realistic audio for complex scenes, rare audio classes, and even unseen audio types, indicating its potential in TTA tasks.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Placed in Conversational and speech RAG, one of 15 directions in Retrieval-augmented generation. Builds on 21 of its 26 references that are in the map. First posted on arXiv 14 Sep 2023.

Builds on

21 of 26 references.

References · 21

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 707
  6. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Modelscited by 592
  7. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 563
  8. 2023Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Modelscited by 323
  9. 2022AudioGen: Textually Guided Audio Generationcited by 292
  10. 2022Masked Autoencoders that Listencited by 271
  11. 2022Diffsound: Discrete Diffusion Model for Text-to-sound Generationcited by 256
  12. 2022Re-Imagen: Retrieval-Augmented Text-to-Image Generatorcited by 208
9 more
  1. 2023Text-to-Audio Generation using Instruction-Tuned LLM and Latent Diffusion Modelcited by 145
  2. 2017PixelSNAIL: An Improved Autoregressive Generative Modelcited by 137
  3. 2021Taming Visually Guided Sound Generationcited by 136
  4. 2022KNN-Diffusion: Image Generation via Large-Scale Retrievalcited by 135
  5. 2023WavJourney: Compositional Audio Creation with Large Language Modelscited by 27
  6. 2021Conditional Sound Generation Using Neural Discrete Time-Frequency Representation Learningcited by 24
  7. 2022Memory-Driven Text-to-Image Generationcited by 10
  8. 2023Leveraging Pre-trained AudioLDM for Sound Generation: A Benchmark Studycited by 10
  9. 2023Text-Driven Foley Sound Generation With Latent Diffusion Modelcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 6; 2026: 6.

23 papers.

Sort:

By area:

  1. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  2. 2024A Comprehensive Survey of Hallucination in Large Language, Image, Video and Audio Foundation Modelscited by 98
  3. 2024A Survey on Retrieval-Augmented Text Generation for Large Language Modelscited by 77
  4. 2025VideoRAG: Retrieval-Augmented Generation over Video Corpuscited by 55
  5. 2024Scaling Transformers for Low-Bitrate High-Quality Speech Codingcited by 48
  6. 2024RAG and RAU: A Survey on Retrieval-Augmented Language Model in Natural Language Processingcited by 25
  7. 2024FlowSep: Language-Queried Sound Separation with Rectified Flow Matchingcited by 21
  8. 2024Retrieval Augmented End-to-End Spoken Dialog Modelscited by 20
  9. 2024T-CLAP: Temporal-Enhanced Contrastive Language-Audio Pretrainingcited by 18
  10. 2024A Survey of Sustainability in Large Language Models: Applications, Economics, and Challengescited by 18
  11. 2025MeanAudio: Fast and Faithful Text-to-Audio Generation with Mean Flowscited by 14
  12. 2024A Survey of Deep Learning Audio Generation Methodscited by 5
11 more
  1. 2025Retrieval-Augmented Dialogue Knowledge Aggregation for Expressive Conversational Speech Synthesiscited by 4
  2. 2024Leveraging Retrieval Augment Approach for Multimodal Emotion Recognition Under Missing Modalitiescited by 3
  3. 2025DreamAudio: Customized Text-to-Audio Generation with Diffusion Modelscited by 2
  4. 2025AutoStyle-TTS: Retrieval-Augmented Generation based Automatic Style Matching Text-to-Speech Synthesiscited by 1
  5. 2026MedSpeak: A Knowledge Graph-Aided ASR Error Correction Framework for Spoken Medical QAcited by 1
  6. 2026FlowSep 2: Self-Supervised Flow Matching for Language-Queried Audio Source Separationcited by 1
  7. 2025InfiniteAudio: Infinite-Length Audio Generation with Consistencycited by 0
  8. 2026SoundWeaver: Semantic Warm-Starting for Text-to-Audio Diffusion Servingcited by 0
  9. 2026Mental Damage: Caption Poisoning Attacks on Retrieval-Augmented Text-to-Music Generationcited by 0
  10. 2026AI-Based Sound Effect Generation: A Narrative Review of Generative Models Across Input Modalitiescited by 0
  11. 2026RAG-Audio: Retrieval-Augmented Generation for Faithful Brain-to-Audio Reconstructioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Audiobox TTA-RAG: Improving Zero-Shot and Few-Shot Text-To-Audio with Retrieval-Augmented Generation
  2. 2026UNITE-AUDIO: Joint Learning of Continuous Tokenization and Latent Flow Matching for Text-to-Audio Generation
  3. 2024Retrieval Augmented Generation in Prompt-based Text-to-Speech Synthesis with Context-Aware Contrastive Language-Audio Pretraining
  4. 2024Enhancing Retrieval-Augmented Audio Captioning with Generation-Assisted Multimodal Querying and Progressive Learning
  5. 2024DiffATR: Diffusion-based Generative Modeling for Audio-Text Retrieval
  6. 2023VoiceLDM: Text-to-Speech with Environmental Context
  7. 2023RECAP: Retrieval-Augmented Audio Captioning
  8. 2024Auffusion: Leveraging the Power of Diffusion and Large Language Models for Text-to-Audio Generation