mlsanityData to 14 Oct 2026

VoiceDiT: Dual-Condition Diffusion Transformer for Environment-Aware Speech Synthesis

arXiv 2412.19259 ↗First posted 26 Dec 2024eess.AS cs.SDcited by 10

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

We present VoiceDiT, a multi-modal generative model for producing environment-aware speech and audio from text and visual prompts. While aligning speech with text is crucial for intelligible speech, achieving this alignment in noisy conditions remains a significant and underexplored challenge in the field. To address this, we present a novel audio generation pipeline named VoiceDiT. This pipeline includes three key components: (1) the creation of a large-scale synthetic speech dataset for pre-training and a refined real-world speech dataset for fine-tuning, (2) the Dual-DiT, a model designed to efficiently preserve aligned speech information while accurately reflecting environmental conditions, and (3) a diffusion-based Image-to-Audio Translator that allows the model to bridge the gap between audio and image, facilitating the generation of environmental sound that aligns with the multi-modal prompts. Extensive experimental results demonstrate that VoiceDiT outperforms previous models on real-world datasets, showcasing significant improvements in both audio quality and modality integration.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 43 of its 48 references that are in the map. First posted on arXiv 26 Dec 2024.

Builds on

43 of 48 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024AudioLCM: Text-to-Audio Generation with Latent Consistency Models1 Jun
  2. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models5 Mar
  3. 2024Auffusion: Leveraging the Power of Diffusion and Large Language Models for Text-to-Audio Generation2 Jan
  4. 2023Audiobox: Unified Audio Generation with Natural Language Prompts25 Dec
  5. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining10 Aug
  6. 2023Text-to-Audio Generation using Instruction-Tuned LLM and Latent Diffusion Model24 Apr
  7. 2023Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models30 Jan
  8. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Models29 Jan
  9. 2022Diffsound: Discrete Diffusion Model for Text-to-sound Generation20 Jul
  10. 2021Guided-TTS: A Diffusion Model for Text-to-Speech via Classifier Guidance23 Nov
  11. 2021Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech13 May

Other references · 32

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  6. 2022Scalable Diffusion Models with Transformerscited by 6,054
  7. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  8. 2020Longformer: The Long-Document Transformercited by 3,413
  9. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,333
  10. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  11. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  12. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
20 more
  1. 2018VoxCeleb2: Deep Speaker Recognitioncited by 875
  2. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 864
  3. 2017VoxCeleb: a large-scale speaker identification datasetcited by 732
  4. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 719
  5. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 649
  6. 2022AudioGen: Textually Guided Audio Generationcited by 295
  7. 2020Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Searchcited by 273
  8. 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 267
  9. 2018Fr'echet Audio Distance: A Metric for Evaluating Music Enhancement Algorithmscited by 225
  10. 2024Fast Timing-Conditioned Latent Audio Diffusioncited by 181
  11. 2021Taming Visually Guided Sound Generationcited by 136
  12. 2023UniAudio: An Audio Foundation Model Toward Universal Audio Generationcited by 127
  13. 2023LibriTTS-R: A Restored Multi-Speaker Text-to-Speech Corpuscited by 101
  14. 2022I Hear Your True Colors: Image Guided Audio Generationcited by 92
  15. 2023V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Modelscited by 65
  16. 2023VoiceLDM: Text-to-Speech with Environmental Contextcited by 23
  17. 2024ESPnet-SPK: full pipeline speaker embedding toolkit with reproducible recipes, self-supervised front-ends, and off-the-shelf modelscited by 22
  18. 2024FreGrad: Lightweight and Fast Frequency-aware Diffusion Vocodercited by 11
  19. 2022Metric Learning for User-defined Keyword Spottingcited by 7
  20. 2023CrossSpeech: Speaker-independent Acoustic Representation for Cross-lingual Speech Synthesiscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 6.

10 papers.

  1. 2025ControlAudio: Tackling Text-Guided, Timing-Indicated and Intelligible Audio Generation via Progressive Diffusion Modelingcited by 11
  2. 2026Inference-Time Scaling for Joint Audio-Video Generationcited by 4
  3. 2025Unsupervised Single-Channel Audio Separation with Diffusion Source Priorscited by 3
  4. 2025Test-Time Augmentation for Pose-invariant Face Recognitioncited by 1
  5. 2026OmniSonic: Towards Universal and Holistic Audio Generation from Video and Textcited by 1
  6. 2025InfiniteAudio: Infinite-Length Audio Generation with Consistencycited by 0
  7. 2026VividVoice: A Unified Framework for Scene-Aware Visually-Driven Speech Synthesiscited by 0
  8. 2026Unified Synthesis of Compositional Speech and Sound from Free-Form Text Promptscited by 0
  9. 2026ImmersiveTTS: Environment-Aware Text-to-Speech with Multimodal Diffusion Transformer and Domain-Specific Representation Alignmentcited by 0
  10. 2026Towards Real-world Environment-aware Zero-shot Text-to-speech Synthesis via Disentangled Audio Infillingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Audio Palette: A Diffusion Transformer with Multi-Signal Conditioning for Controllable Foley Synthesis
  2. 2026VoiceDesigner: Text-to-Voice Generation and Editing via Unified Diffusion Modeling and Data Augmentation
  3. 2025AlignDiT: Multimodal Aligned Diffusion Transformer for Synchronized Speech Generation
  4. 2023A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AI
  5. 2024Visual Echoes: A Simple Unified Transformer for Audio-Visual Generation
  6. 2025Audio Generation Through Score-Based Generative Modeling: Design Principles and Implementation
  7. 2026AlignAudio: A Unified Dual-Alignment Framework for Noise-Robust Speech-Conditioned Generation
  8. 20253MDiT: Unified Tri-Modal Diffusion Transformer for Text-Driven Synchronized Audio-Video Generation