# Cross-speaker Style Transfer with Prosody Bottleneck in Neural Speech Synthesis

arXiv: https://arxiv.org/abs/2107.12562 · HTML: https://mlsanity.com/paper/2107.12562/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 14 references that are in the map.
- First posted on arXiv 27 Jul 2021.

## Builds on (most cited first, 10 of 10)

- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [Tacotron: Towards End-to-End Speech Synthesis](https://mlsanity.com/paper/1703.10135/) (2017)
- [Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis](https://mlsanity.com/paper/1803.09017/) (2018)
- [Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron](https://mlsanity.com/paper/1803.09047/) (2018)
- [Learning latent representations for style control and transfer in end-to-end speech synthesis](https://mlsanity.com/paper/1812.04342/) (2018)
- [Robust and fine-grained prosody control of end-to-end speech synthesis](https://mlsanity.com/paper/1811.02122/) (2018)
- [Fine-grained robust prosody transfer for single-speaker neural text-to-speech](https://mlsanity.com/paper/1907.02479/) (2019)
- [CopyCat: Many-to-Many Fine-Grained Prosody Transfer for Neural Text-to-Speech](https://mlsanity.com/paper/2004.14617/) (2020)
- [Multi-Reference Neural TTS Stylization with Adversarial Cycle Consistency](https://mlsanity.com/paper/1910.11958/) (2019)

## Built on by (most cited first, 9 of 9)

- [EmoSphere++: Emotion-Controllable Zero-Shot Text-to-Speech via Emotion-Adaptive Spherical Vector](https://mlsanity.com/paper/2411.02625/) (2024)
- [Cross-speaker style transfer for text-to-speech using data augmentation](https://mlsanity.com/paper/2202.05083/) (2022)
- [Style-Label-Free: Cross-Speaker Style Transfer by Quantized VAE and Speaker-wise Normalization in Speech Synthesis](https://mlsanity.com/paper/2212.06397/) (2022)
- [Improving Prosody for Cross-Speaker Style Transfer by Semi-Supervised Style Extractor and Hierarchical Modeling in Speech Synthesis](https://mlsanity.com/paper/2303.07711/) (2023)
- [Text-driven Emotional Style Control and Cross-speaker Style Transfer in Neural TTS](https://mlsanity.com/paper/2207.06000/) (2022)
- [Voice Impression Control in Zero-Shot TTS](https://mlsanity.com/paper/2506.05688/) (2025)
- [DiCLET-TTS: Diffusion Model based Cross-lingual Emotion Transfer for Text-to-Speech -- A Study between English and Mandarin](https://mlsanity.com/paper/2309.00883/) (2023)
- [Referee: Towards reference-free cross-speaker style transfer with low-quality data for expressive speech synthesis](https://mlsanity.com/paper/2109.03439/) (2021)
- [An Empirical Study on Learning Latent Representations for Emotional Speech Synthesis](https://mlsanity.com/paper/2606.14922/) (2026)

## Abstract (arXiv)

Cross-speaker style transfer is crucial to the applications of multi-style and expressive speech synthesis at scale. It does not require the target speakers to be experts in expressing all styles and to collect corresponding recordings for model training. However, the performances of existing style transfer methods are still far behind real application needs. The root causes are mainly twofold. Firstly, the style embedding extracted from single reference speech can hardly provide fine-grained and appropriate prosody information for arbitrary text to synthesize. Secondly, in these models the content/text, prosody, and speaker timbre are usually highly entangled, it's therefore not realistic to expect a satisfied result when freely combining these components, such as to transfer speaking style between speakers. In this paper, we propose a cross-speaker style transfer text-to-speech (TTS) model with explicit prosody bottleneck. The prosody bottleneck builds up the kernels accounting for speaking style robustly, and disentangles the prosody from content and speaker timbre, therefore guarantees high quality cross-speaker style transfer. Evaluation result shows the proposed method even achieves on-par performance with source speaker's speaker-dependent (SD) model in objective measurement of prosody, and significantly outperforms the cycle consistency and GMVAE-based baselines in objective and subjective evaluations.