# Sequence to Sequence Neural Speech Synthesis with Prosody Modification Capabilities

arXiv: https://arxiv.org/abs/1909.10302 · HTML: https://mlsanity.com/paper/1909.10302/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Sep 2019.

## Built on by (most cited first, 11 of 11)

- [Exact Prosody Cloning in Zero-Shot Multispeaker Text-to-Speech](https://mlsanity.com/paper/2206.12229/) (2022)
- [Hierarchical prosody modeling and control in non-autoregressive parallel neural TTS](https://mlsanity.com/paper/2110.02952/) (2021)
- [Emphasis control for parallel neural TTS](https://mlsanity.com/paper/2110.03012/) (2021)
- [Prosodic Clustering for Phoneme-level Prosody Control in End-to-End Speech Synthesis](https://mlsanity.com/paper/2111.10177/) (2021)
- [Prosody-controllable spontaneous TTS with neural HMMs](https://mlsanity.com/paper/2211.13533/) (2022)
- [Analysis and Assessment of Controllability of an Expressive Deep Learning-based TTS system](https://mlsanity.com/paper/2103.04097/) (2021)
- [Improved Prosodic Clustering for Multispeaker and Speaker-independent Phoneme-level Prosody Control](https://mlsanity.com/paper/2111.10168/) (2021)
- [Controllable speech synthesis by learning discrete phoneme-level prosodic representations](https://mlsanity.com/paper/2211.16307/) (2022)
- [Karaoker: Alignment-free singing voice synthesis with speech training data](https://mlsanity.com/paper/2204.04127/) (2022)
- [CycleFlow: Purify Information Factors by Cycle Loss](https://mlsanity.com/paper/2110.09928/) (2021)
- [Learning utterance-level representations through token-level acoustic latents prediction for Expressive Speech Synthesis](https://mlsanity.com/paper/2211.00523/) (2022)

## Abstract (arXiv)

Modern sequence to sequence neural TTS systems provide close to natural speech quality. Such systems usually comprise a network converting linguistic/phonetic features sequence to an acoustic features sequence, cascaded with a neural vocoder. The generated speech prosody (i.e. phoneme durations, pitch and loudness) is implicitly present in the acoustic features, being mixed with spectral information. Although the speech sounds natural, its prosody realization is randomly chosen and cannot be easily altered. The prosody control becomes an even more difficult task if no prosodic labeling is present in the training data. Recently, much progress has been achieved in unsupervised speaking style learning and generation, however human inspection is still required after the training for discovery and interpretation of the speaking styles learned by the system. In this work we introduce a fully automatic method that makes the system aware of the prosody and enables sentence-wise speaking pace and expressiveness control on a continuous scale. While being useful by itself in many applications, the proposed prosody control can also improve the overall quality and expressiveness of the synthesized speech, as demonstrated by subjective listening evaluations. We also propose a novel augmented attention mechanism, that facilitates better pace control sensitivity and faster attention convergence.