# Discrete Acoustic Space for an Efficient Sampling in Neural Text-To-Speech

arXiv: https://arxiv.org/abs/2110.12539 · HTML: https://mlsanity.com/paper/2110.12539/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 28 references that are in the map.
- First posted on arXiv 24 Oct 2021.

## Builds on (most cited first, 25 of 25)

- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [An Introduction to Variational Autoencoders](https://mlsanity.com/paper/1906.02691/) (2019)
- [A Primer in BERTology: What we know about how BERT works](https://mlsanity.com/paper/2002.12327/) (2020)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [Jukebox: A Generative Model for Music](https://mlsanity.com/paper/2005.00341/) (2020)
- [Attention-Based Models for Speech Recognition](https://mlsanity.com/paper/1506.07503/) (2015)
- [Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron](https://mlsanity.com/paper/1803.09047/) (2018)
- [Fast Decoding in Sequence Models using Discrete Latent Variables](https://mlsanity.com/paper/1803.03382/) (2018)
- [Hierarchical Generative Modeling for Controllable Speech Synthesis](https://mlsanity.com/paper/1810.07217/) (2018)
- [Learning latent representations for style control and transfer in end-to-end speech synthesis](https://mlsanity.com/paper/1812.04342/) (2018)
- [Fully-hierarchical fine-grained prosody modeling for interpretable speech synthesis](https://mlsanity.com/paper/2002.03785/) (2020)
- [Generating diverse and natural text-to-speech samples using a quantized fine-grained VAE and auto-regressive prosody prior](https://mlsanity.com/paper/2002.03788/) (2020)
- [Universal Neural Vocoding with Parallel WaveNet](https://mlsanity.com/paper/2102.01106/) (2021)
- [Expressive Speech Synthesis via Modeling Expressions with Variational Autoencoder](https://mlsanity.com/paper/1804.02135/) (2018)
- [CAMP: a Two-Stage Approach to Modelling Prosody in Context](https://mlsanity.com/paper/2011.01175/) (2020)
- [Dynamic Prosody Generation for Speech Synthesis using Linguistics-Driven Acoustic Embedding Selection](https://mlsanity.com/paper/1912.00955/) (2019)
- [Hierarchical Multi-Grained Generative Model for Expressive Speech Synthesis](https://mlsanity.com/paper/2009.08474/) (2020)
- [Prosodic Representation Learning and Contextual Sampling for Neural Text-to-Speech](https://mlsanity.com/paper/2011.02252/) (2020)
- [In Other News: A Bi-style Text-to-speech Model for Synthesizing Newscaster Voice with Limited Data](https://mlsanity.com/paper/1904.02790/) (2019)
- [Using generative modelling to produce varied intonation for speech synthesis](https://mlsanity.com/paper/1906.04233/) (2019)
- [A learned conditional prior for the VAE acoustic space of a TTS system](https://mlsanity.com/paper/2106.10229/) (2021)
- [Enhancing audio quality for expressive Neural Text-to-Speech](https://mlsanity.com/paper/2108.06270/) (2021)

## Built on by (most cited first, 3 of 3)

- [Quantised Global Autoencoder: A Holistic Approach to Representing Visual Data](https://mlsanity.com/paper/2407.11913/) (2024)
- [Multidimensional Byte Pair Encoding: Shortened Sequences for Improved Visual Data Generation](https://mlsanity.com/paper/2411.10281/) (2024)
- [TSVer: A Benchmark for Fact Verification Against Time-Series Evidence](https://mlsanity.com/paper/2511.01101/) (2025)

## Abstract (arXiv)

We present a Split Vector Quantized Variational Autoencoder (SVQ-VAE) architecture using a split vector quantizer for NTTS, as an enhancement to the well-known Variational Autoencoder (VAE) and Vector Quantized Variational Autoencoder (VQ-VAE) architectures. Compared to these previous architectures, our proposed model retains the benefits of using an utterance-level bottleneck, while keeping significant representation power and a discretized latent space small enough for efficient prediction from text. We train the model on recordings in the expressive task-oriented dialogues domain and show that SVQ-VAE achieves a statistically significant improvement in naturalness over the VAE and VQ-VAE models. Furthermore, we demonstrate that the SVQ-VAE latent acoustic space is predictable from text, reducing the gap between the standard constant vector synthesis and vocoded recordings by 32%.