# Prosodic Representation Learning and Contextual Sampling for Neural Text-to-Speech

arXiv: https://arxiv.org/abs/2011.02252 · HTML: https://mlsanity.com/paper/2011.02252/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Nov 2020.

## Built on by (most cited first, 10 of 10)

- [ParaTTS: Learning Linguistic and Prosodic Cross-sentence Information in Paragraph-based TTS](https://mlsanity.com/paper/2209.06484/) (2022)
- [A learned conditional prior for the VAE acoustic space of a TTS system](https://mlsanity.com/paper/2106.10229/) (2021)
- [Discrete Acoustic Space for an Efficient Sampling in Neural Text-To-Speech](https://mlsanity.com/paper/2110.12539/) (2021)
- [Controllable speech synthesis by learning discrete phoneme-level prosodic representations](https://mlsanity.com/paper/2211.16307/) (2022)
- [A Study of Modeling Rising Intonation in Cantonese Neural Speech Synthesis](https://mlsanity.com/paper/2208.02189/) (2022)
- [Investigation of Japanese PnG BERT language model in text-to-speech synthesis for pitch accent language](https://mlsanity.com/paper/2212.08321/) (2022)
- [Using multiple reference audios and style embedding constraints for speech synthesis](https://mlsanity.com/paper/2110.04451/) (2021)
- [Controllable Prosody Generation With Partial Inputs](https://mlsanity.com/paper/2303.09446/) (2023)
- [Improving French Synthetic Speech Quality via SSML Prosody Control](https://mlsanity.com/paper/2508.17494/) (2025)
- [Multi-Scale Spectrogram Modelling for Neural Text-to-Speech](https://mlsanity.com/paper/2106.15649/) (2021)

## Abstract (arXiv)

In this paper, we introduce Kathaka, a model trained with a novel two-stage training process for neural speech synthesis with contextually appropriate prosody. In Stage I, we learn a prosodic distribution at the sentence level from mel-spectrograms available during training. In Stage II, we propose a novel method to sample from this learnt prosodic distribution using the contextual information available in text. To do this, we use BERT on text, and graph-attention networks on parse trees extracted from text. We show a statistically significant relative improvement of 13.2% in naturalness over a strong baseline when compared to recordings. We also conduct an ablation study on variations of our sampling technique, and show a statistically significant improvement over the baseline in each case.