# Alternate Endings: Improving Prosody for Incremental Neural TTS with Predicted Future Text Input

arXiv: https://arxiv.org/abs/2102.09914 · HTML: https://mlsanity.com/paper/2102.09914/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 33 references that are in the map.
- First posted on arXiv 19 Feb 2021.

## Builds on (most cited first, 16 of 16)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://mlsanity.com/paper/1904.02882/) (2019)
- [Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram](https://mlsanity.com/paper/1910.11480/) (2019)
- [ESPnet-TTS: Unified, Reproducible, and Integratable Open Source End-to-End Text-to-Speech Toolkit](https://mlsanity.com/paper/1910.10909/) (2019)
- [Semi-Supervised Training for Improving Data Efficiency in End-to-End Speech Synthesis](https://mlsanity.com/paper/1808.10128/) (2018)
- [Incremental Text-to-Speech Synthesis with Prefix-to-Prefix Framework](https://mlsanity.com/paper/1911.02750/) (2019)
- [Simultaneous Speech-to-Speech Translation System with Neural Incremental ASR, MT, and TTS](https://mlsanity.com/paper/2011.04845/) (2020)
- [Towards Transfer Learning for End-to-End Speech Synthesis from Deep Pre-Trained Language Models](https://mlsanity.com/paper/1906.07307/) (2019)
- [Speculative Beam Search for Simultaneous Translation](https://mlsanity.com/paper/1909.05421/) (2019)
- [What the Future Brings: Investigating the Impact of Lookahead for Incremental Neural TTS](https://mlsanity.com/paper/2009.02035/) (2020)
- [Incremental Text to Speech for Neural Sequence-to-Sequence Models using Reinforcement Learning](https://mlsanity.com/paper/2008.03096/) (2020)
- [Incremental Text-to-Speech Synthesis Using Pseudo Lookahead with Large Pretrained Language Model](https://mlsanity.com/paper/2012.12612/) (2020)
- [Learning Dynamic Author Representations with Temporal Language Models](https://mlsanity.com/paper/1909.04985/) (2019)
- [Incremental Machine Speech Chain Towards Enabling Listening while Speaking in Real-time](https://mlsanity.com/paper/2011.02126/) (2020)

## Built on by (most cited first, 7 of 7)

- [A Survey on Neural Speech Synthesis](https://mlsanity.com/paper/2106.15561/) (2021)
- [Direct Simultaneous Speech-to-Speech Translation with Variational Monotonic Multihead Attention](https://mlsanity.com/paper/2110.08250/) (2021)
- [Review of end-to-end speech synthesis technology based on deep learning](https://mlsanity.com/paper/2104.09995/) (2021)
- [From Start to Finish: Latency Reduction Strategies for Incremental Speech Synthesis in Simultaneous Speech-to-Speech Translation](https://mlsanity.com/paper/2110.08214/) (2021)
- [Low-Latency Incremental Text-to-Speech Synthesis with Distilled Context Prediction Network](https://mlsanity.com/paper/2109.10724/) (2021)
- [Efficient Incremental Text-to-Speech on GPUs](https://mlsanity.com/paper/2211.13939/) (2022)
- [Llama-VITS: Enhancing TTS Synthesis with Semantic Awareness](https://mlsanity.com/paper/2404.06714/) (2024)

## Abstract (arXiv)

The prosody of a spoken word is determined by its surrounding context. In incremental text-to-speech synthesis, where the synthesizer produces an output before it has access to the complete input, the full context is often unknown which can result in a loss of naturalness in the synthesized speech. In this paper, we investigate whether the use of predicted future text can attenuate this loss. We compare several test conditions of next future word: (a) unknown (zero-word), (b) language model predicted, (c) randomly predicted and (d) ground-truth. We measure the prosodic features (pitch, energy and duration) and find that predicted text provides significant improvements over a zero-word lookahead, but only slight gains over random-word lookahead. We confirm these results with a perceptive test.