# A Study of Modeling Rising Intonation in Cantonese Neural Speech Synthesis

arXiv: https://arxiv.org/abs/2208.02189 · HTML: https://mlsanity.com/paper/2208.02189/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 32 references that are in the map.
- First posted on arXiv 3 Aug 2022.

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech](https://mlsanity.com/paper/2106.06103/) (2021)
- [Tacotron: Towards End-to-End Speech Synthesis](https://mlsanity.com/paper/1703.10135/) (2017)
- [Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis](https://mlsanity.com/paper/1803.09017/) (2018)
- [Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron](https://mlsanity.com/paper/1803.09047/) (2018)
- [Hierarchical Generative Modeling for Controllable Speech Synthesis](https://mlsanity.com/paper/1810.07217/) (2018)
- [Multi-band MelGAN: Faster Waveform Generation for High-Quality Text-to-Speech](https://mlsanity.com/paper/2005.05106/) (2020)
- [Learning latent representations for style control and transfer in end-to-end speech synthesis](https://mlsanity.com/paper/1812.04342/) (2018)
- [ESPnet-TTS: Unified, Reproducible, and Integratable Open Source End-to-End Text-to-Speech Toolkit](https://mlsanity.com/paper/1910.10909/) (2019)
- [Efficiently Trainable Text-to-Speech System Based on Deep Convolutional Networks with Guided Attention](https://mlsanity.com/paper/1710.08969/) (2017)
- [PnG BERT: Augmented BERT on Phonemes and Graphemes for Neural TTS](https://mlsanity.com/paper/2103.15060/) (2021)
- [Improving Prosody Modelling with Cross-Utterance BERT Embeddings for End-to-end Speech Synthesis](https://mlsanity.com/paper/2011.05161/) (2020)
- [CAMP: a Two-Stage Approach to Modelling Prosody in Context](https://mlsanity.com/paper/2011.01175/) (2020)
- [Dynamic Prosody Generation for Speech Synthesis using Linguistics-Driven Acoustic Embedding Selection](https://mlsanity.com/paper/1912.00955/) (2019)
- [Speech BERT Embedding For Improving Prosody in Neural TTS](https://mlsanity.com/paper/2106.04312/) (2021)
- [Exploiting Syntactic Features in a Parsed Tree to Improve End-to-End TTS](https://mlsanity.com/paper/1904.04764/) (2019)
- [Prosodic Representation Learning and Contextual Sampling for Neural Text-to-Speech](https://mlsanity.com/paper/2011.02252/) (2020)
- [Prosodic Clustering for Phoneme-level Prosody Control in End-to-End Speech Synthesis](https://mlsanity.com/paper/2111.10177/) (2021)
- [Syntactic representation learning for neural network based TTS with syntactic parse tree traversal](https://mlsanity.com/paper/2012.06971/) (2020)

## Built on by (most cited first, 2 of 2)

- [QI-TTS: Questioning Intonation Control for Emotional Speech Synthesis](https://mlsanity.com/paper/2303.07682/) (2023)
- [Using Deepfake Technologies for Word Emphasis Detection](https://mlsanity.com/paper/2305.07791/) (2023)

## Abstract (arXiv)

In human speech, the attitude of a speaker cannot be fully expressed only by the textual content. It has to come along with the intonation. Declarative questions are commonly used in daily Cantonese conversations, and they are usually uttered with rising intonation. Vanilla neural text-to-speech (TTS) systems are not capable of synthesizing rising intonation for these sentences due to the loss of semantic information. Though it has become more common to complement the systems with extra language models, their performance in modeling rising intonation is not well studied. In this paper, we propose to complement the Cantonese TTS model with a BERT-based statement/question classifier. We design different training strategies and compare their performance. We conduct our experiments on a Cantonese corpus named CanTTS. Empirical results show that the separate training approach obtains the best generalization performance and feasibility.