mlsanityData to 14 Oct 2026

Style Description based Text-to-Speech with Conditional Prosodic Layer Normalization based Diffusion GAN

arXiv 2310.18169 ↗First posted 27 Oct 2023cs.SD cs.CL eess.AScited by 0

Not on the map. Why?

Abstract (arXiv)

In this paper, we present a Diffusion GAN based approach (Prosodic Diff-TTS) to generate the corresponding high-fidelity speech based on the style description and content text as an input to generate speech samples within only 4 denoising steps. It leverages the novel conditional prosodic layer normalization to incorporate the style embeddings into the multi head attention based phoneme encoder and mel spectrogram decoder based generator architecture to generate the speech. The style embedding is generated by fine tuning the pretrained BERT model on auxiliary tasks such as pitch, speaking speed, emotion,gender classifications. We demonstrate the efficacy of our proposed architecture on multi-speaker LibriTTS and PromptSpeech datasets, using multiple quantitative metrics that measure generated accuracy and MOS.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 18 of its 22 references that are in the map. First posted on arXiv 27 Oct 2023.

Builds on

18 of 22 references.

References · 18

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  4. 2016Least Squares Generative Adversarial Networkscited by 1,001
  5. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 804
  6. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 717
  7. 2021Tackling the Generative Learning Trilemma with Denoising Diffusion GANscited by 516
  8. 2015Deep Speech 2: End-to-End Speech Recognition in English and Mandarincited by 451
  9. 2015Autoencoding beyond pixels using a learned similarity metriccited by 407
  10. 2022FastDiff: A Fast Conditional Diffusion Model for High-Quality Speech Synthesiscited by 136
  11. 2022PromptTTS: Controllable Text-to-Speech with Text Descriptionscited by 110
  12. 2017Deep Voice: Real-time Neural Text-to-Speechcited by 83
6 more
  1. 2017Deep Voice 3: Scaling Text-to-Speech with Convolutional Sequence Learningcited by 67
  2. 2022DiffGAN-TTS: High-Fidelity and Efficient Text-to-Speech with Denoising Diffusion GANscited by 43
  3. 2019Token-Level Ensemble Distillation for Grapheme-to-Phoneme Conversioncited by 26
  4. 2020VocGAN: A High-Fidelity Real-time Vocoder with a Hierarchically-nested Adversarial Networkcited by 26
  5. 2020Speaking Speed Control of End-to-End Speech Synthesis using Sentence-Level Conditioningcited by 8
  6. 2022Unsupervised word-level prosody tagging for controllable speech synthesiscited by 7

Built on by

Not cited yet.