# Prosody Analysis of Audiobooks

arXiv: https://arxiv.org/abs/2310.06930 · HTML: https://mlsanity.com/paper/2310.06930/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 55 references that are in the map.
- First posted on arXiv 10 Oct 2023.

## Builds on (most cited first, 30 of 33)

- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [MPNet: Masked and Permuted Pre-training for Language Understanding](https://mlsanity.com/paper/2004.09297/) (2020)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [Stanza: A Python Natural Language Processing Toolkit for Many Human Languages](https://mlsanity.com/paper/2003.07082/) (2020)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [Tacotron: Towards End-to-End Speech Synthesis](https://mlsanity.com/paper/1703.10135/) (2017)
- [Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis](https://mlsanity.com/paper/1803.09017/) (2018)
- [Constituency Parsing with a Self-Attentive Encoder](https://mlsanity.com/paper/1805.01052/) (2018)
- [Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron](https://mlsanity.com/paper/1803.09047/) (2018)
- [FastPitch: Parallel Text-to-speech with Pitch Prediction](https://mlsanity.com/paper/2006.06873/) (2020)
- [Hierarchical Generative Modeling for Controllable Speech Synthesis](https://mlsanity.com/paper/1810.07217/) (2018)
- [Deep Voice: Real-time Neural Text-to-Speech](https://mlsanity.com/paper/1702.07825/) (2017)
- [Deep Voice 2: Multi-Speaker Neural Text-to-Speech](https://mlsanity.com/paper/1705.08947/) (2017)
- [Deep Voice 3: Scaling Text-to-Speech with Convolutional Sequence Learning](https://mlsanity.com/paper/1710.07654/) (2017)
- [Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modeling](https://mlsanity.com/paper/2010.04301/) (2020)
- [Parallel Tacotron: Non-Autoregressive and Controllable TTS](https://mlsanity.com/paper/2010.11439/) (2020)
- ["Let Your Characters Tell Their Story": A Dataset for Character-Centric Narrative Understanding](https://mlsanity.com/paper/2109.05438/) (2021)
- [Wave-Tacotron: Spectrogram-free end-to-end text-to-speech synthesis](https://mlsanity.com/paper/2011.03568/) (2020)
- [DurIAN: Duration Informed Attention Network For Multimodal Synthesis](https://mlsanity.com/paper/1909.01700/) (2019)
- [Parallel Tacotron 2: A Non-Autoregressive Neural TTS Model with Differentiable Duration Modeling](https://mlsanity.com/paper/2103.14574/) (2021)
- [LightSpeech: Lightweight and Fast Text to Speech with Neural Architecture Search](https://mlsanity.com/paper/2102.04040/) (2021)
- [Screenplay Summarization Using Latent Narrative Structure](https://mlsanity.com/paper/2004.12727/) (2020)
- [From Once Upon a Time to Happily Ever After: Tracking Emotions in Novels and Fairy Tales](https://mlsanity.com/paper/1309.5909/) (2013)
- [SpeedySpeech: Efficient Neural Speech Synthesis](https://mlsanity.com/paper/2008.03802/) (2020)
- [Modelling Suspense in Short Stories as Uncertainty Reduction over Neural Representation](https://mlsanity.com/paper/2004.14905/) (2020)
- [Memory and Knowledge Augmented Language Models for Inferring Salience in Long-Form Stories](https://mlsanity.com/paper/2109.03754/) (2021)
- [Emotional Prosody Control for Speech Generation](https://mlsanity.com/paper/2111.04730/) (2021)
- [Multi-reference Tacotron by Intercross Training for Style Disentangling,Transfer and Control in Speech Synthesis](https://mlsanity.com/paper/1904.02373/) (2019)
- [Modeling Event Salience in Narratives via Barthes' Cardinal Functions](https://mlsanity.com/paper/2011.01785/) (2020)
- [What time is it? Temporal Analysis of Novels](https://mlsanity.com/paper/2011.04124/) (2020)

## Built on by (most cited first, 3 of 3)

- [MultiActor-Audiobook: Zero-Shot Audiobook Generation with Faces and Voices of Multiple Speakers](https://mlsanity.com/paper/2505.13082/) (2025)
- [Balalaika: Data-Centric, Prosody-Aware Annotation Pipeline for Russian Speech](https://mlsanity.com/paper/2507.13563/) (2025)
- [Audio-Based Understanding of Audiobook Narration Appeal](https://mlsanity.com/paper/2607.02473/) (2026)

## Abstract (arXiv)

Recent advances in text-to-speech have made it possible to generate natural-sounding audio from text. However, audiobook narrations involve dramatic vocalizations and intonations by the reader, with greater reliance on emotions, dialogues, and descriptions in the narrative. Using our dataset of 93 aligned book-audiobook pairs, we present improved models for prosody prediction properties (pitch, volume, and rate of speech) from narrative text using language modeling. Our predicted prosody attributes correlate much better with human audiobook readings than results from a state-of-the-art commercial TTS system: our predicted pitch shows a higher correlation with human reading for 22 out of the 24 books, while our predicted volume attribute proves more similar to human reading for 23 out of the 24 books. Finally, we present a human evaluation study to quantify the extent that people prefer prosody-enhanced audiobook readings over commercial text-to-speech systems.