# A Human-in-the-Loop Approach to Improving Cross-Text Prosody Transfer

arXiv: https://arxiv.org/abs/2406.06601 · HTML: https://mlsanity.com/paper/2406.06601/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 28 references that are in the map.
- First posted on arXiv 6 Jun 2024.

## Builds on (most cited first, 16 of 16)

- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [A Survey of Human-in-the-loop for Machine Learning](https://mlsanity.com/paper/2108.00941/) (2021)
- [Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron](https://mlsanity.com/paper/1803.09047/) (2018)
- [Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehension](https://mlsanity.com/paper/2002.00293/) (2020)
- [Gibbs Sampling with People](https://mlsanity.com/paper/2008.02595/) (2020)
- [CHiVE: Varying Prosody in Speech Synthesis with a Linguistically Driven Dynamic Hierarchical Conditional Variational Network](https://mlsanity.com/paper/1905.07195/) (2019)
- [CopyCat: Many-to-Many Fine-Grained Prosody Transfer for Neural Text-to-Speech](https://mlsanity.com/paper/2004.14617/) (2020)
- [Controllable neural text-to-speech synthesis using intuitive prosodic features](https://mlsanity.com/paper/2009.06775/) (2020)
- [Emotion controllable speech synthesis using emotion-unlabeled dataset with the assistance of cross-domain speech emotion recognition](https://mlsanity.com/paper/2010.13350/) (2020)
- [Deep Encoder-Decoder Models for Unsupervised Learning of Controllable Speech Synthesis](https://mlsanity.com/paper/1807.11470/) (2018)
- [Daft-Exprt: Cross-Speaker Prosody Transfer on Any Text for Expressive Speech Synthesis](https://mlsanity.com/paper/2108.02271/) (2021)
- [Ctrl-P: Temporal Control of Prosodic Variation for Speech Synthesis](https://mlsanity.com/paper/2106.08352/) (2021)
- [Exploring emotional prototypes in a high dimensional TTS latent space](https://mlsanity.com/paper/2105.01891/) (2021)
- [Human-in-the-loop Speaker Adaptation for DNN-based Multi-speaker TTS](https://mlsanity.com/paper/2206.10256/) (2022)
- [Do Prosody Transfer Models Transfer Prosody?](https://mlsanity.com/paper/2303.04289/) (2023)

## Abstract (arXiv)

Text-To-Speech (TTS) prosody transfer models can generate varied prosodic renditions, for the same text, by conditioning on a reference utterance. These models are trained with a reference that is identical to the target utterance. But when the reference utterance differs from the target text, as in cross-text prosody transfer, these models struggle to separate prosody from text, resulting in reduced perceived naturalness. To address this, we propose a Human-in-the-Loop (HitL) approach. HitL users adjust salient correlates of prosody to make the prosody more appropriate for the target text, while maintaining the overall reference prosodic effect. Human adjusted renditions maintain the reference prosody while being rated as more appropriate for the target text 57.8% of the time. Our analysis suggests that limited user effort suffices for these improvements, and that closeness in the latent reference space is not a reliable prosodic similarity metric for the cross-text condition.