mlsanityData to 7 Oct 2026

AudioGen: Textually Guided Audio Generation

arXiv 2209.15352 ↗First posted 30 Sep 2022cs.SD cs.CL cs.LGcited by 292

Not on the map. Why?

Cited by 80 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We tackle the problem of generating audio samples conditioned on descriptive text captions. In this work, we propose AaudioGen, an auto-regressive generative model that generates audio samples conditioned on text inputs. AudioGen operates on a learnt discrete audio representation. The task of text-to-audio generation poses multiple challenges. Due to the way audio travels through a medium, differentiating ``objects'' can be a difficult task (e.g., separating multiple people simultaneously speaking). This is further complicated by real-world recording conditions (e.g., background noise, reverberation, etc.). Scarce text annotations impose another constraint, limiting the ability to scale models. Finally, modeling high-fidelity audio requires encoding audio at high sampling rate, leading to extremely long sequences. To alleviate the aforementioned challenges we propose an augmentation technique that mixes different audio samples, driving the model to internally learn to separate multiple sources. We curated 10 datasets containing different types of audio and text annotations to handle the scarcity of text-audio data points. For faster inference, we explore the use of multi-stream modeling, allowing the use of shorter sequences while maintaining a similar bitrate and perceptual quality. We apply classifier-free guidance to improve adherence to text. Comparing to the evaluated baselines, AudioGen outperforms over both objective and subjective metrics. Finally, we explore the ability of the proposed method to generate audio continuation conditionally and unconditionally. Samples: https://felixkreuk.github.io/audiogen

Key facts. Cited by 292 papers in the map (data through 7 Oct 2026). Builds on 54 of its 69 references that are in the map. First posted on arXiv 30 Sep 2022.

Builds on

54 of 69 references.

References · 54

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  8. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  9. 2022Classifier-Free Diffusion Guidancecited by 5,717
  10. 2016Layer Normalizationcited by 5,579
  11. 2017Neural Discrete Representation Learningcited by 4,650
  12. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
38 more
  1. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  2. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  3. 2020Longformer: The Long-Document Transformercited by 3,362
  4. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  6. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  7. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  8. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  9. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,295
  10. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  11. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  12. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,061
  13. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  14. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
  15. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  16. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
  17. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  18. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  19. 2019Clotho: An Audio Captioning Datasetcited by 354
  20. 2019MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesiscited by 351
  21. 2020FSD50K: An Open Dataset of Human-Labeled Sound Eventscited by 339
  22. 2019Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogramcited by 337
  23. 2021Generative Spoken Language Modeling from Raw Audiocited by 336
  24. 2022Masked Autoencoders that Listencited by 271
  25. 2022Diffsound: Discrete Diffusion Model for Text-to-sound Generationcited by 256
  26. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  27. 2021Efficient Training of Audio Transformers with Patchoutcited by 222
  28. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 177
  29. 2021Direct speech-to-speech translation with discrete unitscited by 148
  30. 2022Generative Spoken Dialogue Language Modelingcited by 125
  31. 2021Textless Speech-to-Speech Translation on Real Datacited by 121
  32. 2021Text-Free Prosody-Aware Generative Spoken Language Modelingcited by 119
  33. 2017Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Datacited by 92
  34. 2020ViSQOL v3: An Open Source Production Ready Objective Speech and Audio Metriccited by 76
  35. 2021UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Datacited by 76
  36. 2020Self-Supervised Contrastive Learning for Unsupervised Phoneme Segmentationcited by 43
  37. 2022Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentationcited by 43
  38. 2020A Spectral Energy Distance for Parallel Speech Synthesiscited by 42

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 72; 2024: 96; 2025: 79; 2026: 39.

292 papers.

Sort:

By area:

  1. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Modelscited by 592
  2. 2023Simple and Controllable Music Generationcited by 480
  3. 2023MusicLM: Generating Music From Textcited by 442
  4. 2023Shap-E: Generating Conditional 3D Implicit Functionscited by 379
  5. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretrainingcited by 343
  6. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  7. 2023Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Modelscited by 323
  8. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  9. 2024Discrete Flow Matchingcited by 301
  10. 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 266
  11. 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizerscited by 233
  12. 2022MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generationcited by 225
38 more
  1. 2024Stable Audio Opencited by 192
  2. 2023Noise2Music: Text-conditioned Music Generation with Diffusion Modelscited by 173
  3. 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 166
  4. 2023Text-to-Audio Generation using Instruction-Tuned LLM and Latent Diffusion Modelcited by 145
  5. 2023Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Modelscited by 133
  6. 2024SAFREE: Training-Free and Adaptive Guard for Safe Text-to-Image And Video Generationcited by 125
  7. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  8. 2024WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modelingcited by 124
  9. 2023Audiobox: Unified Audio Generation with Natural Language Promptscited by 120
  10. 2024VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wildcited by 118
  11. 2024Large Language Models for Time Series: A Surveycited by 113
  12. 2023SoundStorm: Efficient Parallel Audio Generationcited by 111
  13. 2024Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Alignerscited by 111
  14. 2025Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokenscited by 109
  15. 2023Textually Pretrained Speech Language Modelscited by 99
  16. 2023MusicLDM: Enhancing Novelty in Text-to-Music Generation Using Beat-Synchronous Mixup Strategiescited by 94
  17. 2023Red teaming ChatGPT via Jailbreaking: Bias, Robustness, Reliability and Toxicitycited by 86
  18. 2023AUDIT: Audio Editing by Following Instructions with Latent Diffusion Modelscited by 81
  19. 2023LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPTcited by 77
  20. 2024Detecting Multimedia Generated by Large AI Models: A Surveycited by 73
  21. 2023Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptationcited by 68
  22. 2024Proactive Detection of Voice Cloning with Localized Watermarkingcited by 67
  23. 2024Natural language guidance of high-fidelity text-to-speech with synthetic annotationscited by 66
  24. 2023V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Modelscited by 65
  25. 2023Multi-Source Diffusion Models for Simultaneous Music Generation and Separationcited by 63
  26. 2023Large AI Models in Health Informatics: Applications, Challenges, and the Futurecited by 61
  27. 2024We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMscited by 61
  28. 2024Auffusion: Leveraging the Power of Diffusion and Large Language Models for Text-to-Audio Generationcited by 54
  29. 2023Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Biascited by 51
  30. 2024Temporally Aligned Audio for Video with Autoregressioncited by 51
  31. 2023Efficient Neural Music Generationcited by 50
  32. 2024Masked Audio Generation using a Single Non-Autoregressive Transformercited by 50
  33. 2024Read, Watch and Scream! Sound Generation from Text and Videocited by 48
  34. 2024OmniFlow: Any-to-Any Generation with Multi-Modal Rectified Flowscited by 45
  35. 2023Separate Anything You Describecited by 43
  36. 2024SemantiCodec: An Ultra Low Bitrate Semantic Audio Codec for General Soundcited by 42
  37. 2023JEN-1: Text-Guided Universal Music Generation with Omnidirectional Diffusion Modelscited by 41
  38. 2023Anticipatory Music Transformercited by 35

50 most cited of 292 All 292 (JSON)