# Speech BERT Embedding For Improving Prosody in Neural TTS

arXiv: https://arxiv.org/abs/2106.04312 · HTML: https://mlsanity.com/paper/2106.04312/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Jun 2021.

## Built on by (most cited first, 11 of 11)

- [A Survey on Neural Speech Synthesis](https://mlsanity.com/paper/2106.15561/) (2021)
- [StyleTTS: A Style-Based Generative Model for Natural and Diverse Text-to-Speech Synthesis](https://mlsanity.com/paper/2205.15439/) (2022)
- [EE-TTS: Emphatic Expressive TTS with Linguistic Information](https://mlsanity.com/paper/2305.12107/) (2023)
- [Exploring speech style spaces with language models: Emotional TTS without emotion labels](https://mlsanity.com/paper/2405.11413/) (2024)
- [Controllable speech synthesis by learning discrete phoneme-level prosodic representations](https://mlsanity.com/paper/2211.16307/) (2022)
- [A Study of Modeling Rising Intonation in Cantonese Neural Speech Synthesis](https://mlsanity.com/paper/2208.02189/) (2022)
- [Investigation of Japanese PnG BERT language model in text-to-speech synthesis for pitch accent language](https://mlsanity.com/paper/2212.08321/) (2022)
- [Cross-Utterance Conditioned VAE for Non-Autoregressive Text-to-Speech](https://mlsanity.com/paper/2205.04120/) (2022)
- [Representation of perceived prosodic similarity of conversational feedback](https://mlsanity.com/paper/2505.13268/) (2025)
- [PRESENT: Zero-Shot Text-to-Prosody Control](https://mlsanity.com/paper/2408.06827/) (2024)
- [Word-wise intonation model for cross-language TTS systems](https://mlsanity.com/paper/2409.20374/) (2024)

## Abstract (arXiv)

This paper presents a speech BERT model to extract embedded prosody information in speech segments for improving the prosody of synthesized speech in neural text-to-speech (TTS). As a pre-trained model, it can learn prosody attributes from a large amount of speech data, which can utilize more data than the original training data used by the target TTS. The embedding is extracted from the previous segment of a fixed length in the proposed BERT. The extracted embedding is then used together with the mel-spectrogram to predict the following segment in the TTS decoder. Experimental results obtained by the Transformer TTS show that the proposed BERT can extract fine-grained, segment-level prosody, which is complementary to utterance-level prosody to improve the final prosody of the TTS speech. The objective distortions measured on a single speaker TTS are reduced between the generated speech and original recordings. Subjective listening tests also show that the proposed approach is favorably preferred over the TTS without the BERT prosody embedding module, for both in-domain and out-of-domain applications. For Microsoft professional, single/multiple speakers and the LJ Speaker in the public database, subjective preference is similarly confirmed with the new BERT prosody embedding. TTS demo audio samples are in https://judy44chen.github.io/TTSSpeechBERT/.