# A Discourse-level Multi-scale Prosodic Model for Fine-grained Emotion Analysis

arXiv: https://arxiv.org/abs/2309.11849 · HTML: https://mlsanity.com/paper/2309.11849/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 34 references that are in the map.
- First posted on arXiv 21 Sep 2023.

## Builds on (most cited first, 25 of 25)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [Revisiting Pre-Trained Models for Chinese Natural Language Processing](https://mlsanity.com/paper/2004.13922/) (2020)
- [Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis](https://mlsanity.com/paper/1803.09017/) (2018)
- [Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron](https://mlsanity.com/paper/1803.09047/) (2018)
- [FastPitch: Parallel Text-to-speech with Pitch Prediction](https://mlsanity.com/paper/2006.06873/) (2020)
- [LPCNet: Improving Neural Speech Synthesis Through Linear Prediction](https://mlsanity.com/paper/1810.11846/) (2018)
- [Deep Voice: Real-time Neural Text-to-Speech](https://mlsanity.com/paper/1702.07825/) (2017)
- [Robust and fine-grained prosody control of end-to-end speech synthesis](https://mlsanity.com/paper/1811.02122/) (2018)
- [Mellotron: Multispeaker expressive voice synthesis by conditioning on rhythm, pitch and global style tokens](https://mlsanity.com/paper/1910.11997/) (2019)
- [Fully-hierarchical fine-grained prosody modeling for interpretable speech synthesis](https://mlsanity.com/paper/2002.03785/) (2020)
- [Predicting Expressive Speaking Style From Text In End-To-End Speech Synthesis](https://mlsanity.com/paper/1808.01410/) (2018)
- [Fine-grained Emotion Strength Transfer, Control and Prediction for Emotional Speech Synthesis](https://mlsanity.com/paper/2011.08477/) (2020)
- [Multiple Instance Learning Networks for Fine-Grained Sentiment Analysis](https://mlsanity.com/paper/1711.09645/) (2017)
- [Towards Multi-Scale Style Control for Expressive Speech Synthesis](https://mlsanity.com/paper/2104.03521/) (2021)
- [Discourse-Aware Semantic Self-Attention for Narrative Reading Comprehension](https://mlsanity.com/paper/1908.10721/) (2019)
- [Multi-Reference Neural TTS Stylization with Adversarial Cycle Consistency](https://mlsanity.com/paper/1910.11958/) (2019)
- [Prosody Transfer in Neural Text to Speech Using Global Pitch and Loudness Features](https://mlsanity.com/paper/1911.09645/) (2019)
- [Controllable Neural Prosody Synthesis](https://mlsanity.com/paper/2008.03388/) (2020)
- [Hierarchical Multi-Grained Generative Model for Expressive Speech Synthesis](https://mlsanity.com/paper/2009.08474/) (2020)
- [Multi-speaker Multi-style Text-to-speech Synthesis With Single-speaker Single-style Training Data Scenarios](https://mlsanity.com/paper/2112.12743/) (2021)

## Built on by (most cited first, 1 of 1)

- [MultiActor-Audiobook: Zero-Shot Audiobook Generation with Faces and Voices of Multiple Speakers](https://mlsanity.com/paper/2505.13082/) (2025)

## Abstract (arXiv)

This paper explores predicting suitable prosodic features for fine-grained emotion analysis from the discourse-level text. To obtain fine-grained emotional prosodic features as predictive values for our model, we extract a phoneme-level Local Prosody Embedding sequence (LPEs) and a Global Style Embedding as prosodic speech features from the speech with the help of a style transfer model. We propose a Discourse-level Multi-scale text Prosodic Model (D-MPM) that exploits multi-scale text to predict these two prosodic features. The proposed model can be used to analyze better emotional prosodic features and thus guide the speech synthesis model to synthesize more expressive speech. To quantitatively evaluate the proposed model, we contribute a new and large-scale Discourse-level Chinese Audiobook (DCA) dataset with more than 13,000 utterances annotated sequences to evaluate the proposed model. Experimental results on the DCA dataset show that the multi-scale text information effectively helps to predict prosodic features, and the discourse-level text improves both the overall coherence and the user experience. More interestingly, although we aim at the synthesis effect of the style transfer model, the synthesized speech by the proposed text prosodic analysis model is even better than the style transfer from the original speech in some user evaluation indicators.