# DenoiSpeech: Denoising Text to Speech with Frame-Level Noise Modeling

arXiv: https://arxiv.org/abs/2012.09547 · HTML: https://mlsanity.com/paper/2012.09547/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Dec 2020.

## Built on by (most cited first, 14 of 14)

- [A Survey on Neural Speech Synthesis](https://mlsanity.com/paper/2106.15561/) (2021)
- [AdaSpeech: Adaptive Text to Speech for Custom Voice](https://mlsanity.com/paper/2103.00993/) (2021)
- [Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias](https://mlsanity.com/paper/2306.03509/) (2023)
- [fairseq S^2: A Scalable and Integrable Speech Synthesis Toolkit](https://mlsanity.com/paper/2109.06912/) (2021)
- [A Vector Quantized Approach for Text to Speech Synthesis on Real-World Spontaneous Speech](https://mlsanity.com/paper/2302.04215/) (2023)
- [Style-Label-Free: Cross-Speaker Style Transfer by Quantized VAE and Speaker-wise Normalization in Speech Synthesis](https://mlsanity.com/paper/2212.06397/) (2022)
- [Improving Prosody for Cross-Speaker Style Transfer by Semi-Supervised Style Extractor and Hierarchical Modeling in Speech Synthesis](https://mlsanity.com/paper/2303.07711/) (2023)
- [Pheme: Efficient and Conversational Speech Generation](https://mlsanity.com/paper/2401.02839/) (2024)
- [Text-To-Speech Synthesis In The Wild](https://mlsanity.com/paper/2409.08711/) (2024)
- [EE-TTS: Emphatic Expressive TTS with Linguistic Information](https://mlsanity.com/paper/2305.12107/) (2023)
- [Noise-robust zero-shot text-to-speech synthesis conditioned on self-supervised speech-representation model with adapters](https://mlsanity.com/paper/2401.05111/) (2024)
- [Fine-grained Noise Control for Multispeaker Speech Synthesis](https://mlsanity.com/paper/2204.05070/) (2022)
- [Preliminary study on using vector quantization latent spaces for TTS/VC systems with consistent performance](https://mlsanity.com/paper/2106.13479/) (2021)
- [AlignAudio: A Unified Dual-Alignment Framework for Noise-Robust Speech-Conditioned Generation](https://mlsanity.com/paper/2610.14697/) (2026)

## Abstract (arXiv)

While neural-based text to speech (TTS) models can synthesize natural and intelligible voice, they usually require high-quality speech data, which is costly to collect. In many scenarios, only noisy speech of a target speaker is available, which presents challenges for TTS model training for this speaker. Previous works usually address the challenge using two methods: 1) training the TTS model using the speech denoised with an enhancement model; 2) taking a single noise embedding as input when training with noisy speech. However, they usually cannot handle speech with real-world complicated noise such as those with high variations along time. In this paper, we develop DenoiSpeech, a TTS system that can synthesize clean speech for a speaker with noisy speech data. In DenoiSpeech, we handle real-world noisy speech by modeling the fine-grained frame-level noise with a noise condition module, which is jointly trained with the TTS model. Experimental results on real-world data show that DenoiSpeech outperforms the previous two methods by 0.31 and 0.66 MOS respectively.