# Controllable neural text-to-speech synthesis using intuitive prosodic features

arXiv: https://arxiv.org/abs/2009.06775 · HTML: https://mlsanity.com/paper/2009.06775/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Sep 2020.

## Built on by (most cited first, 19 of 19)

- [Controllable Data Generation by Deep Learning: A Review](https://mlsanity.com/paper/2207.09542/) (2022)
- [SOMOS: The Samsung Open MOS Dataset for the Evaluation of Neural Text-to-Speech Synthesis](https://mlsanity.com/paper/2204.03040/) (2022)
- [Exact Prosody Cloning in Zero-Shot Multispeaker Text-to-Speech](https://mlsanity.com/paper/2206.12229/) (2022)
- [BERT, can HE predict contrastive focus? Predicting and controlling prominence in neural TTS using a language model](https://mlsanity.com/paper/2207.01718/) (2022)
- [Hierarchical prosody modeling and control in non-autoregressive parallel neural TTS](https://mlsanity.com/paper/2110.02952/) (2021)
- [Emphasis control for parallel neural TTS](https://mlsanity.com/paper/2110.03012/) (2021)
- [Ctrl-P: Temporal Control of Prosodic Variation for Speech Synthesis](https://mlsanity.com/paper/2106.08352/) (2021)
- [Prosodic Clustering for Phoneme-level Prosody Control in End-to-End Speech Synthesis](https://mlsanity.com/paper/2111.10177/) (2021)
- [Prosody-controllable spontaneous TTS with neural HMMs](https://mlsanity.com/paper/2211.13533/) (2022)
- [Zero-shot Voice Conversion via Self-supervised Prosody Representation Learning](https://mlsanity.com/paper/2110.14422/) (2021)
- [FluentEditor: Text-based Speech Editing by Considering Acoustic and Prosody Consistency](https://mlsanity.com/paper/2309.11725/) (2023)
- [Analysis and Assessment of Controllability of an Expressive Deep Learning-based TTS system](https://mlsanity.com/paper/2103.04097/) (2021)
- [Improving multi-speaker TTS prosody variance with a residual encoder and normalizing flows](https://mlsanity.com/paper/2106.05762/) (2021)
- [Improved Prosodic Clustering for Multispeaker and Speaker-independent Phoneme-level Prosody Control](https://mlsanity.com/paper/2111.10168/) (2021)
- [Controllable speech synthesis by learning discrete phoneme-level prosodic representations](https://mlsanity.com/paper/2211.16307/) (2022)
- [Inkorrect: Online Handwriting Spelling Correction](https://mlsanity.com/paper/2202.13794/) (2022)
- [Speaker Adaption with Intuitive Prosodic Features for Statistical Parametric Speech Synthesis](https://mlsanity.com/paper/2203.00951/) (2022)
- [Multi-Scale Spectrogram Modelling for Neural Text-to-Speech](https://mlsanity.com/paper/2106.15649/) (2021)
- [A Human-in-the-Loop Approach to Improving Cross-Text Prosody Transfer](https://mlsanity.com/paper/2406.06601/) (2024)

## Abstract (arXiv)

Modern neural text-to-speech (TTS) synthesis can generate speech that is indistinguishable from natural speech. However, the prosody of generated utterances often represents the average prosodic style of the database instead of having wide prosodic variation. Moreover, the generated prosody is solely defined by the input text, which does not allow for different styles for the same sentence. In this work, we train a sequence-to-sequence neural network conditioned on acoustic speech features to learn a latent prosody space with intuitive and meaningful dimensions. Experiments show that a model conditioned on sentence-wise pitch, pitch range, phone duration, energy, and spectral tilt can effectively control each prosodic dimension and generate a wide variety of speaking styles, while maintaining similar mean opinion score (4.23) to our Tacotron baseline (4.26).