# Unsupervised word-level prosody tagging for controllable speech synthesis

arXiv: https://arxiv.org/abs/2202.07200 · HTML: https://mlsanity.com/paper/2202.07200/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 27 references that are in the map.
- First posted on arXiv 15 Feb 2022.

## Builds on (most cited first, 15 of 15)

- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis](https://mlsanity.com/paper/1910.06711/) (2019)
- [Hierarchical Generative Modeling for Controllable Speech Synthesis](https://mlsanity.com/paper/1810.07217/) (2018)
- [Robust and fine-grained prosody control of end-to-end speech synthesis](https://mlsanity.com/paper/1811.02122/) (2018)
- [Fully-hierarchical fine-grained prosody modeling for interpretable speech synthesis](https://mlsanity.com/paper/2002.03785/) (2020)
- [Generating diverse and natural text-to-speech samples using a quantized fine-grained VAE and auto-regressive prosody prior](https://mlsanity.com/paper/2002.03788/) (2020)
- [Fine-grained robust prosody transfer for single-speaker neural text-to-speech](https://mlsanity.com/paper/1907.02479/) (2019)
- [Expressive Speech Synthesis via Modeling Expressions with Variational Autoencoder](https://mlsanity.com/paper/1804.02135/) (2018)
- [Semi-Supervised Generative Modeling for Controllable Speech Synthesis](https://mlsanity.com/paper/1910.01709/) (2019)
- [Hierarchical Prosody Modeling for Non-Autoregressive Speech Synthesis](https://mlsanity.com/paper/2011.06465/) (2020)
- [Hierarchical Multi-Grained Generative Model for Expressive Speech Synthesis](https://mlsanity.com/paper/2009.08474/) (2020)
- [Phone-Level Prosody Modelling with GMM-Based MDN for Diverse and Controllable Speech Synthesis](https://mlsanity.com/paper/2105.13086/) (2021)
- [Ctrl-P: Temporal Control of Prosodic Variation for Speech Synthesis](https://mlsanity.com/paper/2106.08352/) (2021)
- [Prosodic Clustering for Phoneme-level Prosody Control in End-to-End Speech Synthesis](https://mlsanity.com/paper/2111.10177/) (2021)
- [Rich Prosody Diversity Modelling with Phone-level Mixture Density Network](https://mlsanity.com/paper/2102.00851/) (2021)

## Built on by (most cited first, 7 of 7)

- [PromptTTS: Controllable Text-to-Speech with Text Descriptions](https://mlsanity.com/paper/2211.12171/) (2022)
- [EmoDiff: Intensity Controllable Emotional Text-to-Speech with Soft-Label Guidance](https://mlsanity.com/paper/2211.09496/) (2022)
- [SC VALL-E: Style-Controllable Zero-Shot Text to Speech Synthesizer](https://mlsanity.com/paper/2307.10550/) (2023)
- [StoryTTS: A Highly Expressive Text-to-Speech Dataset with Rich Textual Expressiveness Annotations](https://mlsanity.com/paper/2404.14946/) (2024)
- [Expressive TTS Driven by Natural Language Prompts Using Few Human Annotations](https://mlsanity.com/paper/2311.01260/) (2023)
- [Predicting phoneme-level prosody latents using AR and flow-based Prior Networks for expressive speech synthesis](https://mlsanity.com/paper/2211.01327/) (2022)
- [Style Description based Text-to-Speech with Conditional Prosodic Layer Normalization based Diffusion GAN](https://mlsanity.com/paper/2310.18169/) (2023)

## Abstract (arXiv)

Although word-level prosody modeling in neural text-to-speech (TTS) has been investigated in recent research for diverse speech synthesis, it is still challenging to control speech synthesis manually without a specific reference. This is largely due to lack of word-level prosody tags. In this work, we propose a novel approach for unsupervised word-level prosody tagging with two stages, where we first group the words into different types with a decision tree according to their phonetic content and then cluster the prosodies using GMM within each type of words separately. This design is based on the assumption that the prosodies of different type of words, such as long or short words, should be tagged with different label sets. Furthermore, a TTS system with the derived word-level prosody tags is trained for controllable speech synthesis. Experiments on LJSpeech show that the TTS model trained with word-level prosody tags not only achieves better naturalness than a typical FastSpeech2 model, but also gains the ability to manipulate word-level prosody.