# Exploring emotional prototypes in a high dimensional TTS latent space

arXiv: https://arxiv.org/abs/2105.01891 · HTML: https://mlsanity.com/paper/2105.01891/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 19 references that are in the map.
- First posted on arXiv 5 May 2021.

## Builds on (most cited first, 7 of 7)

- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [Tacotron: Towards End-to-End Speech Synthesis](https://mlsanity.com/paper/1703.10135/) (2017)
- [Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis](https://mlsanity.com/paper/1803.09017/) (2018)
- [Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron](https://mlsanity.com/paper/1803.09047/) (2018)
- [Flowtron: an Autoregressive Flow-based Generative Network for Text-to-Speech Synthesis](https://mlsanity.com/paper/2005.05957/) (2020)
- [Mellotron: Multispeaker expressive voice synthesis by conditioning on rhythm, pitch and global style tokens](https://mlsanity.com/paper/1910.11997/) (2019)
- [Gibbs Sampling with People](https://mlsanity.com/paper/2008.02595/) (2020)

## Built on by (most cited first, 6 of 6)

- [Getting aligned on representational alignment](https://mlsanity.com/paper/2310.13018/) (2023)
- [Characterizing Similarities and Divergences in Conversational Tones in Humans and LLMs by Sampling with People](https://mlsanity.com/paper/2406.04278/) (2024)
- [Probing Perceptual Priors of MLLMs via Gibbs Sampling with Interpretable Generative Controls](https://mlsanity.com/paper/2608.27727/) (2026)
- [Intercategorical Label Interpolation for Emotional Face Generation with Conditional Generative Adversarial Networks](https://mlsanity.com/paper/2204.12237/) (2022)
- [Daisy-TTS: Simulating Wider Spectrum of Emotions via Prosody Embedding Decomposition](https://mlsanity.com/paper/2402.14523/) (2024)
- [A Human-in-the-Loop Approach to Improving Cross-Text Prosody Transfer](https://mlsanity.com/paper/2406.06601/) (2024)

## Abstract (arXiv)

Recent TTS systems are able to generate prosodically varied and realistic speech. However, it is unclear how this prosodic variation contributes to the perception of speakers' emotional states. Here we use the recent psychological paradigm 'Gibbs Sampling with People' to search the prosodic latent space in a trained GST Tacotron model to explore prototypes of emotional prosody. Participants are recruited online and collectively manipulate the latent space of the generative speech model in a sequentially adaptive way so that the stimulus presented to one group of participants is determined by the response of the previous groups. We demonstrate that (1) particular regions of the model's latent space are reliably associated with particular emotions, (2) the resulting emotional prototypes are well-recognized by a separate group of human raters, and (3) these emotional prototypes can be effectively transferred to new sentences. Collectively, these experiments demonstrate a novel approach to the understanding of emotional speech by providing a tool to explore the relation between the latent space of generative models and human semantics.