mlsanityData to 7 Oct 2026

NoreSpeech: Knowledge Distillation based Conditional Diffusion Model for Noise-robust Expressive TTS

arXiv 2211.02448 ↗First posted 4 Nov 2022cs.SD eess.AScited by 8

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

Expressive text-to-speech (TTS) can synthesize a new speaking style by imiating prosody and timbre from a reference audio, which faces the following challenges: (1) The highly dynamic prosody information in the reference audio is difficult to extract, especially, when the reference audio contains background noise. (2) The TTS systems should have good generalization for unseen speaking styles. In this paper, we present a noise-robust expressive TTS model (NoreSpeech), which can robustly transfer speaking style in a noisy reference utterance to synthesized speech. Specifically, our NoreSpeech includes several components: (1) a novel DiffStyle module, which leverages powerful probabilistic denoising diffusion models to learn noise-agnostic speaking style features from a teacher model by knowledge distillation; (2) a VQ-VAE block, which maps the style features into a controllable quantized latent space for improving the generalization of style transfer; and (3) a straight-forward but effective parameter-free text-style alignment module, which enables NoreSpeech to transfer style to a textual input from a length-mismatched reference utterance. Experiments demonstrate that NoreSpeech is more effective than previous expressive TTS models in noise environments. Audio samples and code are available at: hrefhttp://dongchaoyang.top/NoreSpeechdemo/http://dongchaoyang.top/NoreSpeechdemo/

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 24 of its 36 references that are in the map. First posted on arXiv 4 Nov 2022.

Builds on

24 of 36 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022Diffsound: Discrete Diffusion Model for Text-to-sound Generation20 Jul
  2. 2022Conditional Diffusion Probabilistic Model for Speech Enhancement10 Feb
  3. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesis21 Sep

Other references · 21

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  3. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  4. 2017Neural Discrete Representation Learningcited by 4,650
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  6. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  7. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  8. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
  9. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  10. 2018Additive Margin Softmax for Face Verificationcited by 275
  11. 2018Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesiscited by 257
  12. 2021Meta-StyleSpeech : Multi-Speaker Adaptive Text-to-Speech Generationcited by 93
9 more
  1. 2020Unsupervised Speech Decomposition via Triple Information Bottleneckcited by 87
  2. 2021Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised Representationscited by 81
  3. 2018A multi-device dataset for urban acoustic scene classificationcited by 75
  4. 2020Fully-hierarchical fine-grained prosody modeling for interpretable speech synthesiscited by 41
  5. 2022GenerSpeech: Towards Style Transfer for Generalizable Out-Of-Domain Text-to-Speechcited by 21
  6. 2021STYLER: Style Factor Modeling with Rapidity and Robustness via Speech Decomposition for Expressive and Controllable Neural Text to Speechcited by 18
  7. 2021Towards Multi-Scale Style Control for Expressive Speech Synthesiscited by 18
  8. 2022SATTS: Speaker Attractor Text to Speech, Learning to Speak by Learning to Separatecited by 3
  9. 2022Fine-grained Noise Control for Multispeaker Speech Synthesiscited by 2

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 5.

8 papers.

  1. 2023A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AIcited by 68
  2. 2023On the Design Fundamentals of Diffusion Models: A Surveycited by 59
  3. 2023TextrolSpeech: A Text Style Control Speech Corpus With Codec Language Text-to-Speech Modelscited by 48
  4. 2024SimpleSpeech: Towards Simple and Efficient Text-to-Speech with Scalar Latent Transformer Diffusion Modelscited by 23
  5. 2024SimpleSpeech 2: Towards Simple and Efficient Text-to-Speech with Flow-based Scalar Latent Transformer Diffusion Modelscited by 17
  6. 2024Noise-robust zero-shot text-to-speech synthesis conditioned on self-supervised speech-representation model with adapterscited by 4
  7. 2024A Comprehensive Survey on Diffusion Models and Their Applicationscited by 4
  8. 2024LiteFocus: Accelerated Diffusion Inference for Long Audio Synthesiscited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2024Style Mixture of Experts for Expressive Text-To-Speech Synthesis
  2. 2022Grad-StyleSpeech: Any-speaker Adaptive Text-to-Speech Synthesis with Diffusion Models
  3. 2024StyleTTS-ZS: Efficient High-Quality Zero-Shot Text-to-Speech Synthesis with Distilled Time-Varying Style Diffusion
  4. 2023StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models
  5. 2025DiffCSS: Diverse and Expressive Conversational Speech Synthesis with Diffusion Models
  6. 2023InstructTTS: Modelling Expressive TTS in Discrete Latent Space with Natural Language Style Prompt
  7. 2024On the Semantic Latent Space of Diffusion-Based Text-to-Speech Models
  8. 2024Towards Expressive Zero-Shot Speech Synthesis with Hierarchical Prosody Modeling