Cross-Utterance Conditioned VAE for Speech Generation
Not on the map. Why?
Abstract (arXiv)
Speech synthesis systems powered by neural networks hold promise for multimedia production, but frequently face issues with producing expressive speech and seamless editing. In response, we present the Cross-Utterance Conditioned Variational Autoencoder speech synthesis (CUC-VAE S2) framework to enhance prosody and ensure natural speech generation. This framework leverages the powerful representational capabilities of pre-trained language models and the re-expression abilities of variational autoencoders (VAEs). The core component of the CUC-VAE S2 framework is the cross-utterance CVAE, which extracts acoustic, speaker, and textual features from surrounding sentences to generate context-sensitive prosodic features, more accurately emulating human prosody generation. We further propose two practical algorithms tailored for distinct speech synthesis applications: CUC-VAE TTS for text-to-speech and CUC-VAE SE for speech editing. The CUC-VAE TTS is a direct application of the framework, designed to generate audio with contextual prosody derived from surrounding texts. On the other hand, the CUC-VAE SE algorithm leverages real mel spectrogram sampling conditioned on contextual information, producing audio that closely mirrors real sound and thereby facilitating flexible speech editing based on text such as deletion, insertion, and replacement. Experimental results on the LibriTTS datasets demonstrate that our proposed models significantly enhance speech synthesis and editing, producing more natural and expressive speech.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 35 of its 53 references that are in the map. First posted on arXiv 8 Sep 2023.
Builds on
35 of 53 references.
References · 35
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
- 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
- 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 804
- 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
- 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 717
- 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 649
- 2021Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speechcited by 447
- 2020Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Searchcited by 273
- 2021Diff-TTS: A Denoising Diffusion Model for Text-to-Speechcited by 157
- 2020FastPitch: Parallel Text-to-speech with Pitch Predictioncited by 151
- 2017Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Datacited by 93
23 more
- 2018Hierarchical Generative Modeling for Controllable Speech Synthesiscited by 93
- 2021PnG BERT: Augmented BERT on Phonemes and Graphemes for Neural TTScited by 52
- 2018Robust and fine-grained prosody control of end-to-end speech synthesiscited by 48
- 2020Fully-hierarchical fine-grained prosody modeling for interpretable speech synthesiscited by 41
- 2022A3T: Alignment-Aware Acoustic and Text Pretraining for Speech Synthesis and Editingcited by 40
- 2020Generating diverse and natural text-to-speech samples using a quantized fine-grained VAE and auto-regressive prosody priorcited by 33
- 2021EditSpeech: A Text Based Speech Editing System Using Partial Inference and Bidirectional Fusioncited by 32
- 2021Parallel Tacotron 2: A Non-Autoregressive Neural TTS Model with Differentiable Duration Modelingcited by 31
- 2017Learning Latent Representations for Speech Generation and Transformationcited by 27
- 2020Improving Prosody Modelling with Cross-Utterance BERT Embeddings for End-to-end Speech Synthesiscited by 27
- 2018Expressive Speech Synthesis via Modeling Expressions with Variational Autoencodercited by 22
- 2022ProsoSpeech: Enhancing Prosody With Quantized Vector Pre-training in Text-to-Speechcited by 22
- 2020AlignTTS: Efficient Feed-Forward Text-to-Speech System without Explicit Alignmentcited by 20
- 2020Unified Mandarin TTS Front-end Based on Distilled BERT Modelcited by 18
- 2021Controllable Context-aware Conversational Speech Synthesiscited by 18
- 2022CampNet: Context-Aware Mask Prediction for End-to-End Text-Based Speech Editingcited by 16
- 2020EfficientTTS: An Efficient and High-Quality Text-to-Speech Architecturecited by 14
- 2021Phrase break prediction with bidirectional encoder representations in Japanese text-to-speech synthesiscited by 12
- 2019Towards Transfer Learning for End-to-End Speech Synthesis from Deep Pre-Trained Language Modelscited by 10
- 2019Non-Autoregressive Neural Text-to-Speechcited by 9
- 2020End-to-End Text-to-Speech using Latent Duration based on VQ-VAEcited by 7
- 2021A learned conditional prior for the VAE acoustic space of a TTS systemcited by 4
- 2022Cross-Utterance Conditioned VAE for Non-Autoregressive Text-to-Speechcited by 1
Built on by
Citing papers per year: 2024: 1.
1 paper.