mlsanityData to 14 Oct 2026

UmbraTTS: Adapting Text-to-Speech to Environmental Contexts with Flow Matching

arXiv 2506.09874 ↗First posted 11 Jun 2025cs.SD cs.LG eess.AScited by 2

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

Recent advances in Text-to-Speech (TTS) have enabled highly natural speech synthesis, yet integrating speech with complex background environments remains challenging. We introduce UmbraTTS, a flow-matching based TTS model that jointly generates both speech and environmental audio, conditioned on text and acoustic context. Our model allows fine-grained control over background volume and produces diverse, coherent, and context-aware audio scenes. A key challenge is the lack of data with speech and background audio aligned in natural context. To overcome the lack of paired training data, we propose a self-supervised framework that extracts speech, background audio, and transcripts from unannotated recordings. Extensive evaluations demonstrate that UmbraTTS significantly outperformed existing baselines, producing natural, high-quality, environmentally aware audios.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 29 of its 36 references that are in the map. First posted on arXiv 11 Jun 2025.

Builds on

29 of 36 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025FlowTSE: Target Speaker Extraction with Flow Matching20 May
  2. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching9 Oct
  3. 2024Tango 2: Aligning Diffusion-based Text-to-Audio Generations through Direct Preference Optimization15 Apr
  4. 2023Audiobox: Unified Audio Generation with Natural Language Prompts25 Dec
  5. 2023Matcha-TTS: A fast TTS architecture with conditional flow matching6 Sep
  6. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining10 Aug
  7. 2023A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AI23 Mar
  8. 2023Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models30 Jan
  9. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Models29 Jan

Other references · 20

  1. 2022Scalable Diffusion Models with Transformerscited by 6,054
  2. 2022Flow Matching for Generative Modelingcited by 5,702
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  4. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 719
  5. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 717
  6. 2023ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoderscited by 712
  7. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  8. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 379
  9. 2020FSD50K: An Open Dataset of Human-Labeled Sound Eventscited by 342
  10. 2022AudioGen: Textually Guided Audio Generationcited by 295
  11. 2021YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyonecited by 278
  12. 2018Fr'echet Audio Distance: A Metric for Evaluating Music Enhancement Algorithmscited by 225
8 more
  1. 2024Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generationcited by 204
  2. 2021A Survey on Neural Speech Synthesiscited by 184
  3. 2022NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Qualitycited by 152
  4. 2024E2 TTS: Embarrassingly Easy Fully Non-Autoregressive Zero-Shot TTScited by 152
  5. 2023MossFormer2: Combining Transformer and RNN-Free Recurrent Network for Enhanced Time-Domain Monaural Speech Separationcited by 39
  6. 2023VoiceLDM: Text-to-Speech with Environmental Contextcited by 23
  7. 2020Enhancing Speech Intelligibility in Text-To-Speech Synthesis using Speaking Style Conversioncited by 6
  8. 2024Speak in the Scene: Diffusion-based Acoustic Scene Transfer toward Immersive Speech Generationcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025ControlAudio: Tackling Text-Guided, Timing-Indicated and Intelligible Audio Generation via Progressive Diffusion Modelingcited by 11
  2. 2026ImmersiveTTS: Environment-Aware Text-to-Speech with Multimodal Diffusion Transformer and Domain-Specific Representation Alignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026OmniSonic: Towards Universal and Holistic Audio Generation from Video and Text
  2. 2026Towards Real-world Environment-aware Zero-shot Text-to-speech Synthesis via Disentangled Audio Infilling
  3. 2024VoiceDiT: Dual-Condition Diffusion Transformer for Environment-Aware Speech Synthesis
  4. 2026Reference-Driven Multi-Speaker Audio Scene Generation from In-the-Wild Priors
  5. 2025UniFlow-Audio: Unified Flow Matching for Audio Generation from Omni-Modalities
  6. 2026UNITE-AUDIO: Joint Learning of Continuous Tokenization and Latent Flow Matching for Text-to-Audio Generation
  7. 2026VoxAudio: Vocalized Audio Synthesis via Multi-Reward Autoregressive Flow Matching
  8. 2026UAT: Unified Audio-Text Diffusion for Audio Generation, Editing, and Captioning