mlsanityData to 14 Oct 2026

ImmersiveTTS: Environment-Aware Text-to-Speech with Multimodal Diffusion Transformer and Domain-Specific Representation Alignment

arXiv 2605.30965 ↗First posted 29 May 2026eess.AS cs.AI cs.CLcited by 0

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

Recent advancements in text-guided audio generation have yielded promising results in diverse domains, including sound effects, speech, and music. However, jointly generating speech with environmental audio remains challenging due to the inherent disparities in their acoustic patterns and temporal dynamics. We propose ImmersiveTTS, an environment-aware text-to-speech (TTS) model that generates natural speech seamlessly integrated within environmental contexts by explicitly modeling cross-modal interactions. Our model builds on a multimodal diffusion transformer and fuses transcript-aligned speech latent with text-conditioned environmental context via joint attention. To enhance semantic consistency, we introduce a domain-specific representation alignment objective tailored to environment-aware TTS, leveraging complementary self-supervised representations from speech and audio encoders. Experimental results show that ImmersiveTTS achieves higher naturalness, intelligibility, and audio fidelity than existing approaches across objective metrics and human listening tests.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 43 of its 59 references that are in the map. First posted on arXiv 29 May 2026.

Builds on

43 of 59 references.

Closest ancestors · 14

Same direction, newest first

  1. 2025HunyuanVideo-Foley: Multimodal Diffusion with Representation Alignment for High-Fidelity Foley Audio Generation23 Aug
  2. 2025UmbraTTS: Adapting Text-to-Speech to Environmental Contexts with Flow Matching11 Jun
  3. 2025ACE-Step: A Step Towards Music Generation Foundation Model28 May
  4. 2025Accelerating Diffusion-based Text-to-Speech Model Training with Dual Modality Alignment26 May
  5. 2025FLowHigh: Towards Efficient and High-Quality Audio Super-Resolution with Single-Step Flow Matching9 Jan
  6. 2024TangoFlux: Super Fast and Faithful Text to Audio Generation with Flow Matching and Clap-Ranked Preference Optimization30 Dec
  7. 2024VoiceDiT: Dual-Condition Diffusion Transformer for Environment-Aware Speech Synthesis26 Dec
  8. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching9 Oct
  9. 2024Auffusion: Leveraging the Power of Diffusion and Large Language Models for Text-to-Audio Generation2 Jan
  10. 2023Audiobox: Unified Audio Generation with Natural Language Prompts25 Dec
  11. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining10 Aug
  12. 2023DDDM-VC: Decoupled Denoising Diffusion Models with Disentangled Representation and Prior Mixup for Verified Robust Voice Conversion25 May
2 more
  1. 2023ViT-TTS: Visual Text-to-Speech with Scalable Diffusion Transformer22 May
  2. 2023Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models30 Jan

Other references · 29

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  6. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  7. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
  8. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  9. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  10. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  11. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 804
  12. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 719
17 more
  1. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 717
  2. 2016CNN Architectures for Large-Scale Audio Classificationcited by 704
  3. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 649
  4. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 561
  5. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 347
  6. 2024SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformerscited by 340
  7. 2020Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Searchcited by 273
  8. 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 267
  9. 2024MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesiscited by 134
  10. 2024mHuBERT-147: A Compact Multilingual HuBERT Modelcited by 66
  11. 2023HierSpeech++: Bridging the Gap between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-shot Speech Synthesiscited by 35
  12. 2025ThinkSound: Chain-of-Thought Reasoning in Multimodal Large Language Models for Audio Generation and Editingcited by 34
  13. 2023Self-supervised Audio Teacher-Student Transformer for Both Clip-level and Frame-level Taskscited by 25
  14. 2023VoiceLDM: Text-to-Speech with Environmental Contextcited by 23
  15. 2025MeanAudio: Fast and Faithful Text-to-Audio Generation with Mean Flowscited by 14
  16. 2025USAD: Universal Speech and Audio Representation via Distillationcited by 7
  17. 2024Multi-modal and Multi-scale Spatial Environment Understanding for Immersive Visual Text-to-Speechcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Towards Real-world Environment-aware Zero-shot Text-to-speech Synthesis via Disentangled Audio Infilling
  2. 2024Speak in the Scene: Diffusion-based Acoustic Scene Transfer toward Immersive Speech Generation
  3. 2025AlignDiT: Multimodal Aligned Diffusion Transformer for Synchronized Speech Generation
  4. 2025From Text to Talk: Audio-Language Model Needs Non-Autoregressive Joint Training
  5. 2025Audio Palette: A Diffusion Transformer with Multi-Signal Conditioning for Controllable Foley Synthesis
  6. 2026UniSonate: A Unified Model for Speech, Music, and Sound Effect Generation with Text Instructions
  7. 2026UAT: Unified Audio-Text Diffusion for Audio Generation, Editing, and Captioning
  8. 2026LongCat-AudioDiT: High-Fidelity Diffusion Text-to-Speech in the Waveform Latent Space