# Word-wise intonation model for cross-language TTS systems

arXiv: https://arxiv.org/abs/2409.20374 · HTML: https://mlsanity.com/paper/2409.20374/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 33 references that are in the map.
- First posted on arXiv 30 Sep 2024.

## Builds on (most cited first, 16 of 16)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://mlsanity.com/paper/1904.02882/) (2019)
- [WhisperX: Time-Accurate Speech Transcription of Long-Form Audio](https://mlsanity.com/paper/2303.00747/) (2023)
- [NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models](https://mlsanity.com/paper/2403.03100/) (2024)
- [StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models](https://mlsanity.com/paper/2306.07691/) (2023)
- [PromptTTS: Controllable Text-to-Speech with Text Descriptions](https://mlsanity.com/paper/2211.12171/) (2022)
- [PromptTTS 2: Describing and Generating Voices with Text Prompt](https://mlsanity.com/paper/2309.02285/) (2023)
- [RMVPE: A Robust Model for Vocal Pitch Estimation in Polyphonic Music](https://mlsanity.com/paper/2306.15412/) (2023)
- [Phoneme-Level BERT for Enhanced Prosody of Text-to-Speech with Grapheme Predictions](https://mlsanity.com/paper/2301.08810/) (2023)
- [Speech BERT Embedding For Improving Prosody in Neural TTS](https://mlsanity.com/paper/2106.04312/) (2021)
- [BERT, can HE predict contrastive focus? Predicting and controlling prominence in neural TTS using a language model](https://mlsanity.com/paper/2207.01718/) (2022)
- [Pheme: Efficient and Conversational Speech Generation](https://mlsanity.com/paper/2401.02839/) (2024)
- [PSST! Prosodic Speech Segmentation with Transformers](https://mlsanity.com/paper/2302.01984/) (2023)
- [What does BERT learn about prosody?](https://mlsanity.com/paper/2304.12706/) (2023)
- [EE-TTS: Emphatic Expressive TTS with Linguistic Information](https://mlsanity.com/paper/2305.12107/) (2023)
- [An Empirical Study of Speech Language Models for Prompt-Conditioned Speech Synthesis](https://mlsanity.com/paper/2403.12402/) (2024)

## Abstract (arXiv)

In this paper we propose a word-wise intonation model for Russian language and show how it can be generalized for other languages. The proposed model is suitable for automatic data markup and its extended application to text-to-speech systems. It can also be implemented for an intonation contour modeling by using rule-based algorithms or by predicting contours with language models. The key idea is a partial elimination of the variability connected with different placements of a stressed syllable in a word. It is achieved with simultaneous applying of pitch simplification with a dynamic time warping clustering. The proposed model could be used as a tool for intonation research or as a backbone for prosody description in text-to-speech systems. As the advantage of the model, we show its relations with the existing intonation systems as well as the possibility of using language models for prosody prediction. Finally, we demonstrate some practical evidence of the system robustness to parameter variations.