mlsanityData to 7 Oct 2026

VoiceLDM: Text-to-Speech with Environmental Context

arXiv 2309.13664 ↗First posted 24 Sep 2023eess.AS cs.AI cs.CLcited by 23

Not on the map. Why?

Abstract (arXiv)

This paper presents VoiceLDM, a model designed to produce audio that accurately follows two distinct natural language text prompts: the description prompt and the content prompt. The former provides information about the overall environmental context of the audio, while the latter conveys the linguistic content. To achieve this, we adopt a text-to-audio (TTA) model based on latent diffusion models and extend its functionality to incorporate an additional content prompt as a conditional input. By utilizing pretrained contrastive language-audio pretraining (CLAP) and Whisper, VoiceLDM is trained on large amounts of real-world audio without manual annotations or transcriptions. Additionally, we employ dual classifier-free guidance to further enhance the controllability of VoiceLDM. Experimental results demonstrate that VoiceLDM is capable of generating plausible audio that aligns well with both input conditions, even surpassing the speech intelligibility of the ground truth audio on the AudioCaps test set. Furthermore, we explore the text-to-speech (TTS) and zero-shot text-to-audio capabilities of VoiceLDM and show that it achieves competitive results. Demos and code are available at https://voiceldm.github.io.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 26 of its 29 references that are in the map. First posted on arXiv 24 Sep 2023.

Builds on

26 of 29 references.

References · 26

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2020Denoising Diffusion Implicit Modelscited by 9,655
  5. 2022Classifier-Free Diffusion Guidancecited by 5,717
  6. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  7. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  8. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  9. 2017VoxCeleb: a large-scale speaker identification datasetcited by 714
  10. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
  11. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 707
  12. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
14 more
  1. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Modelscited by 592
  2. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretrainingcited by 343
  3. 2023Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Modelscited by 323
  4. 2022AudioGen: Textually Guided Audio Generationcited by 292
  5. 2022Diffsound: Discrete Diffusion Model for Text-to-sound Generationcited by 256
  6. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 177
  7. 2022NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Qualitycited by 151
  8. 2023Text-to-Audio Generation using Instruction-Tuned LLM and Latent Diffusion Modelcited by 145
  9. 2022PromptTTS: Controllable Text-to-Speech with Text Descriptionscited by 106
  10. 2023InstructTTS: Modelling Expressive TTS in Discrete Latent Space with Natural Language Style Promptcited by 90
  11. 2023Make-An-Audio 2: Temporal-Enhanced Text-to-Audio Generationcited by 90
  12. 2023PromptStyle: Controllable Style Transfer for Text-to-Speech with Natural Language Descriptionscited by 33
  13. 2021Expressive Text-to-Speech using Style Tagcited by 20
  14. 2023Enhanced Controllability of Diffusion Models via Feature Disentanglement and Realism-Enhanced Sampling Methodscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 10; 2026: 8.

23 papers.

Sort:

By area:

  1. 2025MegaTTS 3: Sparse Alignment Enhanced Latent Diffusion Transformer for Zero-Shot Speech Synthesiscited by 39
  2. 2024DiTTo-TTS: Diffusion Transformers for Scalable Text-to-Speech without Domain-Specific Factorscited by 31
  3. 2024Towards Controllable Speech Synthesis in the Era of Large Language Models: A Systematic Surveycited by 22
  4. 2025CoVoMix2: Advancing Zero-Shot Dialogue Generation with Fully Non-Autoregressive Flow Matchingcited by 11
  5. 2025ViSAGe: Video-to-Spatial Audio Generationcited by 11
  6. 2025ControlAudio: Tackling Text-Guided, Timing-Indicated and Intelligible Audio Generation via Progressive Diffusion Modelingcited by 11
  7. 2024VoiceDiT: Dual-Condition Diffusion Transformer for Environment-Aware Speech Synthesiscited by 10
  8. 2025MMMG: a Comprehensive and Reliable Benchmark for Multitask Multimodal Generationcited by 5
  9. 2024Multi-Source Spatial Knowledge Understanding for Immersive Visual Text-to-Speechcited by 4
  10. 2024Multi-modal and Multi-scale Spatial Environment Understanding for Immersive Visual Text-to-Speechcited by 4
  11. 2025Selective Classifier-free Guidance for Zero-shot Text-to-speechcited by 4
  12. 2025UmbraTTS: Adapting Text-to-Speech to Environmental Contexts with Flow Matchingcited by 2
11 more
  1. 2025In-the-wild Audio Spatialization with Flexible Text-guided Localizationcited by 1
  2. 2025MaskVCT: Masked Voice Codec Transformer for Zero-Shot Voice Conversion With Increased Controllability via Multiple Guidancescited by 1
  3. 2026ParaMETA: Towards Learning Disentangled Paralinguistic Speaking Styles Representations from Speechcited by 1
  4. 2026OmniSonic: Towards Universal and Holistic Audio Generation from Video and Textcited by 1
  5. 2026Joint Residual Reweighting for Classifier Free Guidance in Flow-Matching Zero-Shot TTScited by 1
  6. 2025InfiniteAudio: Infinite-Length Audio Generation with Consistencycited by 0
  7. 2026VividVoice: A Unified Framework for Scene-Aware Visually-Driven Speech Synthesiscited by 0
  8. 2026Unified Synthesis of Compositional Speech and Sound from Free-Form Text Promptscited by 0
  9. 2026ImmersiveTTS: Environment-Aware Text-to-Speech with Multimodal Diffusion Transformer and Domain-Specific Representation Alignmentcited by 0
  10. 2026Towards Real-world Environment-aware Zero-shot Text-to-speech Synthesis via Disentangled Audio Infillingcited by 0
  11. 2026Learnable Classifier-Free Guidance Null Embeddings for Enhanced Controllable Speech Synthesiscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026LongCat-AudioDiT: High-Fidelity Diffusion Text-to-Speech in the Waveform Latent Space
  2. 2023Audiobox: Unified Audio Generation with Natural Language Prompts
  3. 2023Retrieval-Augmented Text-to-Audio Generation
  4. 2025From Text to Talk: Audio-Language Model Needs Non-Autoregressive Joint Training
  5. 2024On the Semantic Latent Space of Diffusion-Based Text-to-Speech Models
  6. 2025DIFFA: Large Language Diffusion Models Can Listen and Understand
  7. 2023E3 TTS: Easy End-to-End Diffusion-based Text to Speech
  8. 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers