# Hierarchical prosody modeling and control in non-autoregressive parallel neural TTS

arXiv: https://arxiv.org/abs/2110.02952 · HTML: https://mlsanity.com/paper/2110.02952/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 31 references that are in the map.
- First posted on arXiv 6 Oct 2021.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis](https://mlsanity.com/paper/1803.09017/) (2018)
- [Efficient Neural Audio Synthesis](https://mlsanity.com/paper/1802.08435/) (2018)
- [Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron](https://mlsanity.com/paper/1803.09047/) (2018)
- [Hierarchical Generative Modeling for Controllable Speech Synthesis](https://mlsanity.com/paper/1810.07217/) (2018)
- [End-to-End Adversarial Text-to-Speech](https://mlsanity.com/paper/2006.03575/) (2020)
- [Deep Voice 3: Scaling Text-to-Speech with Convolutional Sequence Learning](https://mlsanity.com/paper/1710.07654/) (2017)
- [Learning latent representations for style control and transfer in end-to-end speech synthesis](https://mlsanity.com/paper/1812.04342/) (2018)
- [Parallel Tacotron: Non-Autoregressive and Controllable TTS](https://mlsanity.com/paper/2010.11439/) (2020)
- [Fully-hierarchical fine-grained prosody modeling for interpretable speech synthesis](https://mlsanity.com/paper/2002.03785/) (2020)
- [CHiVE: Varying Prosody in Speech Synthesis with a Linguistically Driven Dynamic Hierarchical Conditional Variational Network](https://mlsanity.com/paper/1905.07195/) (2019)
- [Controllable neural text-to-speech synthesis using intuitive prosodic features](https://mlsanity.com/paper/2009.06775/) (2020)
- [Hierarchical Prosody Modeling for Non-Autoregressive Speech Synthesis](https://mlsanity.com/paper/2011.06465/) (2020)
- [Investigation of enhanced Tacotron text-to-speech synthesis systems with self-attention for pitch accent language](https://mlsanity.com/paper/1810.11960/) (2018)
- [Sequence to Sequence Neural Speech Synthesis with Prosody Modification Capabilities](https://mlsanity.com/paper/1909.10302/) (2019)
- [Prosodic Prominence and Boundaries in Sequence-to-Sequence Speech Synthesis](https://mlsanity.com/paper/2006.15967/) (2020)
- [On-device neural speech synthesis](https://mlsanity.com/paper/2109.08710/) (2021)
- [Supervised and Unsupervised Approaches for Controlling Narrow Lexical Focus in Sequence-to-Sequence Speech Synthesis](https://mlsanity.com/paper/2101.09940/) (2021)
- [Hierarchical Context-Aware Transformers for Non-Autoregressive Text to Speech](https://mlsanity.com/paper/2106.15144/) (2021)

## Built on by (most cited first, 9 of 9)

- [DiffProsody: Diffusion-based Latent Prosody Generation for Expressive Speech Synthesis with Prosody Conditional Adversarial Training](https://mlsanity.com/paper/2307.16549/) (2023)
- [Prosody-controllable spontaneous TTS with neural HMMs](https://mlsanity.com/paper/2211.13533/) (2022)
- [ADAPTERMIX: Exploring the Efficacy of Mixture of Adapters for Low-Resource TTS Adaptation](https://mlsanity.com/paper/2305.18028/) (2023)
- [Emphasis Rendering for Conversational Text-to-Speech with Multi-modal Multi-scale Context Modeling](https://mlsanity.com/paper/2410.09524/) (2024)
- [Counterfactual Activation Editing for Post-hoc Prosody and Mispronunciation Correction in TTS Models](https://mlsanity.com/paper/2506.00832/) (2025)
- [CoMelSinger: Discrete Token-Based Zero-Shot Singing Synthesis With Structured Melody Control and Guidance](https://mlsanity.com/paper/2509.19883/) (2025)
- [Controllable Accented Text-to-Speech Synthesis](https://mlsanity.com/paper/2209.10804/) (2022)
- [Learning utterance-level representations through token-level acoustic latents prediction for Expressive Speech Synthesis](https://mlsanity.com/paper/2211.00523/) (2022)
- [PRODIS -- a speech database and a phoneme-based language model for the study of predictability effects in Polish](https://mlsanity.com/paper/2404.10112/) (2024)

## Abstract (arXiv)

Neural text-to-speech (TTS) synthesis can generate speech that is indistinguishable from natural speech. However, the synthetic speech often represents the average prosodic style of the database instead of having more versatile prosodic variation. Moreover, many models lack the ability to control the output prosody, which does not allow for different styles for the same text input. In this work, we train a non-autoregressive parallel neural TTS front-end model hierarchically conditioned on both coarse and fine-grained acoustic speech features to learn a latent prosody space with intuitive and meaningful dimensions. Experiments show that a non-autoregressive TTS model hierarchically conditioned on utterance-wise pitch, pitch range, duration, energy, and spectral tilt can effectively control each prosodic dimension, generate a wide variety of speaking styles, and provide word-wise emphasis control, while maintaining equal or better quality to the baseline model.