# Incremental Text to Speech for Neural Sequence-to-Sequence Models using Reinforcement Learning

arXiv: https://arxiv.org/abs/2008.03096 · HTML: https://mlsanity.com/paper/2008.03096/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Aug 2020.

## Built on by (most cited first, 8 of 8)

- [A Survey on Neural Speech Synthesis](https://mlsanity.com/paper/2106.15561/) (2021)
- [A Survey on Deep Reinforcement Learning for Audio-Based Applications](https://mlsanity.com/paper/2101.00240/) (2021)
- [Reinforcement Learning for Generative AI: A Survey](https://mlsanity.com/paper/2308.14328/) (2023)
- [Reinforcement Learning and Bandits for Speech and Language Processing: Tutorial, Review and Outlook](https://mlsanity.com/paper/2210.13623/) (2022)
- [Alternate Endings: Improving Prosody for Incremental Neural TTS with Predicted Future Text Input](https://mlsanity.com/paper/2102.09914/) (2021)
- [From Start to Finish: Latency Reduction Strategies for Incremental Speech Synthesis in Simultaneous Speech-to-Speech Translation](https://mlsanity.com/paper/2110.08214/) (2021)
- [Low-Latency Incremental Text-to-Speech Synthesis with Distilled Context Prediction Network](https://mlsanity.com/paper/2109.10724/) (2021)
- [Efficient Incremental Text-to-Speech on GPUs](https://mlsanity.com/paper/2211.13939/) (2022)

## Abstract (arXiv)

Modern approaches to text to speech require the entire input character sequence to be processed before any audio is synthesised. This latency limits the suitability of such models for time-sensitive tasks like simultaneous interpretation. Interleaving the action of reading a character with that of synthesising audio reduces this latency. However, the order of this sequence of interleaved actions varies across sentences, which raises the question of how the actions should be chosen. We propose a reinforcement learning based framework to train an agent to make this decision. We compare our performance against that of deterministic, rule-based systems. Our results demonstrate that our agent successfully balances the trade-off between the latency of audio generation and the quality of synthesised audio. More broadly, we show that neural sequence-to-sequence models can be adapted to run in an incremental manner.