# Ctrl-P: Temporal Control of Prosodic Variation for Speech Synthesis

arXiv: https://arxiv.org/abs/2106.08352 · HTML: https://mlsanity.com/paper/2106.08352/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 30 references that are in the map.
- First posted on arXiv 15 Jun 2021.

## Builds on (most cited first, 19 of 19)

- [Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations](https://mlsanity.com/paper/1811.12359/) (2018)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis](https://mlsanity.com/paper/1803.09017/) (2018)
- [Efficient Neural Audio Synthesis](https://mlsanity.com/paper/1802.08435/) (2018)
- [Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron](https://mlsanity.com/paper/1803.09047/) (2018)
- [FastPitch: Parallel Text-to-speech with Pitch Prediction](https://mlsanity.com/paper/2006.06873/) (2020)
- [Hierarchical Generative Modeling for Controllable Speech Synthesis](https://mlsanity.com/paper/1810.07217/) (2018)
- [Deep Voice 3: Scaling Text-to-Speech with Convolutional Sequence Learning](https://mlsanity.com/paper/1710.07654/) (2017)
- [Learning to Speak Fluently in a Foreign Language: Multilingual Speech Synthesis and Cross-Language Voice Cloning](https://mlsanity.com/paper/1907.04448/) (2019)
- [Robust and fine-grained prosody control of end-to-end speech synthesis](https://mlsanity.com/paper/1811.02122/) (2018)
- [Fully-hierarchical fine-grained prosody modeling for interpretable speech synthesis](https://mlsanity.com/paper/2002.03785/) (2020)
- [Fine-grained robust prosody transfer for single-speaker neural text-to-speech](https://mlsanity.com/paper/1907.02479/) (2019)
- [CHiVE: Varying Prosody in Speech Synthesis with a Linguistically Driven Dynamic Hierarchical Conditional Variational Network](https://mlsanity.com/paper/1905.07195/) (2019)
- [Controllable neural text-to-speech synthesis using intuitive prosodic features](https://mlsanity.com/paper/2009.06775/) (2020)
- [Deep Encoder-Decoder Models for Unsupervised Learning of Controllable Speech Synthesis](https://mlsanity.com/paper/1807.11470/) (2018)
- [Controllable Neural Prosody Synthesis](https://mlsanity.com/paper/2008.03388/) (2020)
- [Using generative modelling to produce varied intonation for speech synthesis](https://mlsanity.com/paper/1906.04233/) (2019)
- [Prosody Learning Mechanism for Speech Synthesis System Without Text Length Limit](https://mlsanity.com/paper/2008.05656/) (2020)

## Built on by (most cited first, 8 of 8)

- [Exact Prosody Cloning in Zero-Shot Multispeaker Text-to-Speech](https://mlsanity.com/paper/2206.12229/) (2022)
- [Unsupervised word-level prosody tagging for controllable speech synthesis](https://mlsanity.com/paper/2202.07200/) (2022)
- [Prosody-controllable spontaneous TTS with neural HMMs](https://mlsanity.com/paper/2211.13533/) (2022)
- [PoeticTTS -- Controllable Poetry Reading for Literary Studies](https://mlsanity.com/paper/2207.05549/) (2022)
- [Controllable speech synthesis by learning discrete phoneme-level prosodic representations](https://mlsanity.com/paper/2211.16307/) (2022)
- [Counterfactual Activation Editing for Post-hoc Prosody and Mispronunciation Correction in TTS Models](https://mlsanity.com/paper/2506.00832/) (2025)
- [Controllable Prosody Generation With Partial Inputs](https://mlsanity.com/paper/2303.09446/) (2023)
- [A Human-in-the-Loop Approach to Improving Cross-Text Prosody Transfer](https://mlsanity.com/paper/2406.06601/) (2024)

## Abstract (arXiv)

Text does not fully specify the spoken form, so text-to-speech models must be able to learn from speech data that vary in ways not explained by the corresponding text. One way to reduce the amount of unexplained variation in training data is to provide acoustic information as an additional learning signal. When generating speech, modifying this acoustic information enables multiple distinct renditions of a text to be produced. Since much of the unexplained variation is in the prosody, we propose a model that generates speech explicitly conditioned on the three primary acoustic correlates of prosody: F0, energy and duration. The model is flexible about how the values of these features are specified: they can be externally provided, or predicted from text, or predicted then subsequently modified. Compared to a model that employs a variational auto-encoder to learn unsupervised latent features, our model provides more interpretable, temporally-precise, and disentangled control. When automatically predicting the acoustic features from text, it generates speech that is more natural than that from a Tacotron 2 model with reference encoder. Subsequent human-in-the-loop modification of the predicted acoustic features can significantly further increase naturalness.