# Simple and Effective Multi-sentence TTS with Expressive and Coherent Prosody

arXiv: https://arxiv.org/abs/2206.14643 · HTML: https://mlsanity.com/paper/2206.14643/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 33 references that are in the map.
- First posted on arXiv 29 Jun 2022.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [Tacotron: Towards End-to-End Speech Synthesis](https://mlsanity.com/paper/1703.10135/) (2017)
- [Generalized End-to-End Loss for Speaker Verification](https://mlsanity.com/paper/1710.10467/) (2017)
- [Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis](https://mlsanity.com/paper/1806.04558/) (2018)
- [Deep Voice 2: Multi-Speaker Neural Text-to-Speech](https://mlsanity.com/paper/1705.08947/) (2017)
- [Zero-Shot Multi-Speaker Text-To-Speech with State-of-the-art Neural Speaker Embeddings](https://mlsanity.com/paper/1910.10838/) (2019)
- [PnG BERT: Augmented BERT on Phonemes and Graphemes for Neural TTS](https://mlsanity.com/paper/2103.15060/) (2021)
- [MultiSpeech: Multi-Speaker Text to Speech with Transformer](https://mlsanity.com/paper/2006.04664/) (2020)
- [DurIAN: Duration Informed Attention Network For Multimodal Synthesis](https://mlsanity.com/paper/1909.01700/) (2019)
- [Improving Prosody Modelling with Cross-Utterance BERT Embeddings for End-to-end Speech Synthesis](https://mlsanity.com/paper/2011.05161/) (2020)
- [Universal Neural Vocoding with Parallel WaveNet](https://mlsanity.com/paper/2102.01106/) (2021)
- [Towards Transfer Learning for End-to-End Speech Synthesis from Deep Pre-Trained Language Models](https://mlsanity.com/paper/1906.07307/) (2019)
- [Evaluating Long-form Text-to-Speech: Comparing the Ratings of Sentences and Paragraphs](https://mlsanity.com/paper/1909.03965/) (2019)
- [In Other News: A Bi-style Text-to-speech Model for Synthesizing Newscaster Voice with Limited Data](https://mlsanity.com/paper/1904.02790/) (2019)
- [CopyCat2: A Single Model for Multi-Speaker TTS and Many-to-Many Fine-Grained Prosody Transfer](https://mlsanity.com/paper/2206.13443/) (2022)
- [s-Transformer: Segment-Transformer for Robust Neural Speech Synthesis](https://mlsanity.com/paper/2011.08480/) (2020)
- [An Empirical Analysis of the Correlation of Syntax and Prosody](https://mlsanity.com/paper/1806.05900/) (2018)
- [Using previous acoustic context to improve Text-to-Speech synthesis](https://mlsanity.com/paper/2012.03763/) (2020)

## Built on by (most cited first, 5 of 5)

- [DiffProsody: Diffusion-based Latent Prosody Generation for Expressive Speech Synthesis with Prosody Conditional Adversarial Training](https://mlsanity.com/paper/2307.16549/) (2023)
- [PauseSpeech: Natural Speech Synthesis via Pre-trained Language Model and Pause-based Prosody Modeling](https://mlsanity.com/paper/2306.07489/) (2023)
- [Controllable Emphasis with zero data for text-to-speech](https://mlsanity.com/paper/2307.07062/) (2023)
- [A Comparative Analysis of Pretrained Language Models for Text-to-Speech](https://mlsanity.com/paper/2309.01576/) (2023)
- [HaikuS2S: A Cascaded System For Responding In Verse](https://mlsanity.com/paper/2609.23951/) (2026)

## Abstract (arXiv)

Generating expressive and contextually appropriate prosody remains a challenge for modern text-to-speech (TTS) systems. This is particularly evident for long, multi-sentence inputs. In this paper, we examine simple extensions to a Transformer-based FastSpeech-like system, with the goal of improving prosody for multi-sentence TTS. We find that long context, powerful text features, and training on multi-speaker data all improve prosody. More interestingly, they result in synergies. Long context disambiguates prosody, improves coherence, and plays to the strengths of Transformers. Fine-tuning word-level features from a powerful language model, such as BERT, appears to profit from more training data, readily available in a multi-speaker setting. We look into objective metrics on pausing and pacing and perform thorough subjective evaluations for speech naturalness. Our main system, which incorporates all the extensions, achieves consistently strong results, including statistically significant improvements in speech naturalness over all its competitors.