# Cross-Utterance Conditioned VAE for Non-Autoregressive Text-to-Speech

arXiv: https://arxiv.org/abs/2205.04120 · HTML: https://mlsanity.com/paper/2205.04120/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 37 references that are in the map.
- First posted on arXiv 9 May 2022.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://mlsanity.com/paper/1904.02882/) (2019)
- [Tacotron: Towards End-to-End Speech Synthesis](https://mlsanity.com/paper/1703.10135/) (2017)
- [FastPitch: Parallel Text-to-speech with Pitch Prediction](https://mlsanity.com/paper/2006.06873/) (2020)
- [Hierarchical Generative Modeling for Controllable Speech Synthesis](https://mlsanity.com/paper/1810.07217/) (2018)
- [PnG BERT: Augmented BERT on Phonemes and Graphemes for Neural TTS](https://mlsanity.com/paper/2103.15060/) (2021)
- [Robust and fine-grained prosody control of end-to-end speech synthesis](https://mlsanity.com/paper/1811.02122/) (2018)
- [Fully-hierarchical fine-grained prosody modeling for interpretable speech synthesis](https://mlsanity.com/paper/2002.03785/) (2020)
- [Generating diverse and natural text-to-speech samples using a quantized fine-grained VAE and auto-regressive prosody prior](https://mlsanity.com/paper/2002.03788/) (2020)
- [Predicting Expressive Speaking Style From Text In End-To-End Speech Synthesis](https://mlsanity.com/paper/1808.01410/) (2018)
- [Parallel Tacotron 2: A Non-Autoregressive Neural TTS Model with Differentiable Duration Modeling](https://mlsanity.com/paper/2103.14574/) (2021)
- [Improving Prosody Modelling with Cross-Utterance BERT Embeddings for End-to-end Speech Synthesis](https://mlsanity.com/paper/2011.05161/) (2020)
- [Expressive Speech Synthesis via Modeling Expressions with Variational Autoencoder](https://mlsanity.com/paper/1804.02135/) (2018)
- [AlignTTS: Efficient Feed-Forward Text-to-Speech System without Explicit Alignment](https://mlsanity.com/paper/2003.01950/) (2020)
- [Controllable Context-aware Conversational Speech Synthesis](https://mlsanity.com/paper/2106.10828/) (2021)
- [Phrase break prediction with bidirectional encoder representations in Japanese text-to-speech synthesis](https://mlsanity.com/paper/2104.12395/) (2021)
- [Speech BERT Embedding For Improving Prosody in Neural TTS](https://mlsanity.com/paper/2106.04312/) (2021)
- [Towards Transfer Learning for End-to-End Speech Synthesis from Deep Pre-Trained Language Models](https://mlsanity.com/paper/1906.07307/) (2019)
- [Non-Autoregressive Neural Text-to-Speech](https://mlsanity.com/paper/1905.08459/) (2019)
- [End-to-End Text-to-Speech using Latent Duration based on VQ-VAE](https://mlsanity.com/paper/2010.09602/) (2020)
- [A learned conditional prior for the VAE acoustic space of a TTS system](https://mlsanity.com/paper/2106.10229/) (2021)

## Built on by (most cited first, 1 of 1)

- [Cross-Utterance Conditioned VAE for Speech Generation](https://mlsanity.com/paper/2309.04156/) (2023)

## Abstract (arXiv)

Modelling prosody variation is critical for synthesizing natural and expressive speech in end-to-end text-to-speech (TTS) systems. In this paper, a cross-utterance conditional VAE (CUC-VAE) is proposed to estimate a posterior probability distribution of the latent prosody features for each phoneme by conditioning on acoustic features, speaker information, and text features obtained from both past and future sentences. At inference time, instead of the standard Gaussian distribution used by VAE, CUC-VAE allows sampling from an utterance-specific prior distribution conditioned on cross-utterance information, which allows the prosody features generated by the TTS system to be related to the context and is more similar to how humans naturally produce prosody. The performance of CUC-VAE is evaluated via a qualitative listening test for naturalness, intelligibility and quantitative measurements, including word error rates and the standard deviation of prosody attributes. Experimental results on LJ-Speech and LibriTTS data show that the proposed CUC-VAE TTS system improves naturalness and prosody diversity with clear margins.