mlsanityData to 7 Oct 2026

Improving Text-To-Audio Models with Synthetic Captions

arXiv 2406.15487 ↗First posted 18 Jun 2024cs.CL cs.LG cs.SDcited by 19

Not on the map. Why?

Abstract (arXiv)

It is an open challenge to obtain high quality training data, especially captions, for text-to-audio models. Although prior methods have leveraged textittext-only language models to augment and improve captions, such methods have limitations related to scale and coherence between audio and captions. In this work, we propose an audio captioning pipeline that uses an textitaudio language model to synthesize accurate and diverse captions for audio at scale. We leverage this pipeline to produce a dataset of synthetic captions for AudioSet, named textttAF-AudioSet, and then evaluate the benefit of pre-training text-to-audio models on these synthetic captions. Through systematic evaluations on AudioCaps and MusicCaps, we find leveraging our pipeline and synthetic captions leads to significant improvements on audio generation quality, achieving a new textitstate-of-the-art.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 31 of its 38 references that are in the map. First posted on arXiv 18 Jun 2024.

Builds on

31 of 38 references.

References · 31

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  6. 2016Improved Techniques for Training GANscited by 2,938
  7. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  8. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  9. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  10. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  11. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 707
  12. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Modelscited by 592
19 more
  1. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  2. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 563
  3. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
  4. 2023Simple and Controllable Music Generationcited by 480
  5. 2023MusicLM: Generating Music From Textcited by 442
  6. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretrainingcited by 343
  7. 2023Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Modelscited by 323
  8. 2022AudioGen: Textually Guided Audio Generationcited by 292
  9. 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 266
  10. 2022BigVGAN: A Universal Neural Vocoder with Large-Scale Trainingcited by 245
  11. 2023Pengi: An Audio Language Model for Audio Taskscited by 217
  12. 2023Listen, Think, and Understandcited by 206
  13. 2023Any-to-Any Generation via Composable Diffusioncited by 203
  14. 2024Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilitiescited by 183
  15. 2024Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durationscited by 160
  16. 2023Text-to-Audio Generation using Instruction-Tuned LLM and Latent Diffusion Modelcited by 145
  17. 2023Mustango: Toward Controllable Text-to-Music Generationcited by 100
  18. 2024Tango 2: Aligning Diffusion-based Text-to-Audio Generations through Direct Preference Optimizationcited by 97
  19. 2024Auffusion: Leveraging the Power of Diffusion and Large Language Models for Text-to-Audio Generationcited by 54

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 9; 2026: 3.

19 papers.

Sort:

By area:

  1. 2024TangoFlux: Super Fast and Faithful Text to Audio Generation with Flow Matching and Clap-Ranked Preference Optimizationcited by 67
  2. 2024EzAudio: Enhancing Text-to-Audio Generation with Efficient Diffusion Transformercited by 30
  3. 2025KAD: No More FAD! An Effective and Efficient Evaluation Metric for Audio Generationcited by 24
  4. 2024ETTA: Elucidating the Design Space of Text-to-Audio Modelscited by 17
  5. 2025MeanAudio: Fast and Faithful Text-to-Audio Generation with Mean Flowscited by 14
  6. 2024SoloAudio: Target Sound Extraction with Language-oriented Audio Diffusion Transformercited by 11
  7. 2024Synthio: Augmenting Small-Scale Audio Classification Datasets with Synthetic Datacited by 7
  8. 2024Challenge on Sound Scene Synthesis: Evaluating Text-to-Audio Generationcited by 6
  9. 2025IMPACT: Iterative Mask-based Parallel Decoding for Text-to-Audio Generation with Diffusion Modelingcited by 6
  10. 2025DeepAudio-V1:Towards Multi-Modal Multi-Stage End-to-End Video to Speech and Audio Generationcited by 5
  11. 2025MMMG: a Comprehensive and Reliable Benchmark for Multitask Multimodal Generationcited by 5
  12. 2025Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Predictioncited by 5
7 more
  1. 2025AudioMoG: Guiding Audio Generation with Mixture-of-Guidancecited by 5
  2. 2024EmotionCaps: Enhancing Audio Captioning Through Emotion-Augmented Data Generationcited by 1
  3. 2025SynSonic: Augmenting Sound Event Detection through Text-to-Audio Diffusion ControlNet and Effective Sample Filteringcited by 1
  4. 2025Enhance Generation Quality of Flow Matching V2A Model via Multi-Step CoT-Like Guidance and Combined Preference Optimizationcited by 0
  5. 2026Robust Audio-Text Retrieval via Cross-Modal Attention and Hybrid Losscited by 0
  6. 2026Noisy-Channel Minimum Bayes Risk Decodingcited by 0
  7. 2026SonicCaps: Large-Scale Diverse and Fine-Grained Captioning for Improved Audio-Retrievalcited by 0