mlsanityData to 14 Oct 2026

An Investigation of Noise Robustness for Flow-Matching-Based Zero-Shot TTS

arXiv 2406.05699 ↗First posted 9 Jun 2024eess.AS cs.AI eess.SPcited by 4

Not on the map. Why?

Abstract (arXiv)

Recently, zero-shot text-to-speech (TTS) systems, capable of synthesizing any speaker's voice from a short audio prompt, have made rapid advancements. However, the quality of the generated speech significantly deteriorates when the audio prompt contains noise, and limited research has been conducted to address this issue. In this paper, we explored various strategies to enhance the quality of audio generated from noisy audio prompts within the context of flow-matching-based zero-shot TTS. Our investigation includes comprehensive training strategies: unsupervised pre-training with masked speech denoising, multi-speaker detection and DNSMOS-based data filtering on the pre-training data, and fine-tuning with random noise mixing. The results of our experiments demonstrate significant improvements in intelligibility, speaker similarity, and overall audio quality compared to the approach of applying speech enhancement to the audio prompt.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 24 of its 27 references that are in the map. First posted on arXiv 9 Jun 2024.

Builds on

24 of 27 references.

References · 24

  1. 2022Flow Matching for Generative Modelingcited by 5,702
  2. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  3. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  4. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
  5. 2015MUSAN: A Music, Speech, and Noise Corpuscited by 425
  6. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 379
  7. 2022BigVGAN: A Universal Neural Vocoder with Large-Scale Trainingcited by 251
  8. 2020DNSMOS: A Non-Intrusive Perceptual Objective Speech Quality metric to evaluate Noise Suppressorscited by 241
  9. 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizerscited by 235
  10. 2021A Survey on Neural Speech Synthesiscited by 184
  11. 2022NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Qualitycited by 152
  12. 2023Audiobox: Unified Audio Generation with Natural Language Promptscited by 120
12 more
  1. 2023SoundStorm: Efficient Parallel Audio Generationcited by 114
  2. 2023A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AIcited by 68
  3. 2023ICASSP 2023 Deep Noise Suppression Challengecited by 65
  4. 2023SpeechX: Neural Codec Language Model as a Versatile Speech Transformercited by 64
  5. 2023Generative Pre-training for Speech with Flow Matchingcited by 55
  6. 2023Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Biascited by 51
  7. 2023E3 TTS: Easy End-to-End Diffusion-based Text to Speechcited by 30
  8. 2022How Bad Are Artifacts?: Analyzing the Impact of Speech Enhancement Errors on ASRcited by 28
  9. 2024Making Flow-Matching-Based Zero-Shot Text-to-Speech Laugh as You Likecited by 11
  10. 2021Human Listening and Live Captioning: Multi-Task Training for Speech Enhancementcited by 8
  11. 2024Noise-robust zero-shot text-to-speech synthesis conditioned on self-supervised speech-representation model with adapterscited by 4
  12. 2022Real-Time Joint Personalized Speech Enhancement and Acoustic Echo Cancellationcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 2.

4 papers.

  1. 2026OmniVoice: Towards Omnilingual Zero-Shot Text-to-Speech with Diffusion Language Modelscited by 25
  2. 2024Laugh Now Cry Later: Controlling Time-Varying Emotional States of Flow-Matching-Based Zero-Shot Text-to-Speechcited by 19
  3. 2025High-Order Matching for One-Step Shortcut Diffusion Modelscited by 15
  4. 2026Towards Real-world Environment-aware Zero-shot Text-to-speech Synthesis via Disentangled Audio Infillingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025DiFlow-TTS: Compact and Low-Latency Zero-Shot Text-to-Speech with Discrete Flow Matching
  2. 2025Flamed-TTS: Flow Matching Attention-Free Models for Efficient Generating and Dynamic Pacing Zero-shot Text-to-Speech
  3. 2025Towards Lightweight and Stable Zero-shot TTS with Self-distilled Representation Disentanglement
  4. 2025OZSpeech: One-step Zero-shot Speech Synthesis with Learned-Prior-Conditioned Flow Matching
  5. 2025Vox-Evaluator: Enhancing Stability and Fidelity for Zero-shot TTS with A Multi-Level Evaluator
  6. 2025ZipVoice: Fast and High-Quality Zero-Shot Text-to-Speech with Flow Matching
  7. 2026RobustSpeechFlow: Learning Robust Text-to-Speech Trajectories via Augmentation-based Contrastive Flow Matching
  8. 2024E2 TTS: Embarrassingly Easy Fully Non-Autoregressive Zero-Shot TTS