# A Vector Quantized Approach for Text to Speech Synthesis on Real-World Spontaneous Speech

arXiv: https://arxiv.org/abs/2302.04215 · HTML: https://mlsanity.com/paper/2302.04215/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 39 references that are in the map.
- First posted on arXiv 8 Feb 2023.

## Builds on (most cited first, 29 of 29)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://mlsanity.com/paper/2108.12409/) (2021)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech](https://mlsanity.com/paper/2106.06103/) (2021)
- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audio](https://mlsanity.com/paper/2106.06909/) (2021)
- [vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations](https://mlsanity.com/paper/1910.05453/) (2019)
- [Generative Spoken Language Modeling from Raw Audio](https://mlsanity.com/paper/2102.01192/) (2021)
- [Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search](https://mlsanity.com/paper/2005.11129/) (2020)
- [Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis](https://mlsanity.com/paper/1803.09017/) (2018)
- [Dawn of the transformer era in speech emotion recognition: closing the valence gap](https://mlsanity.com/paper/2203.07378/) (2022)
- [Monotonic Chunkwise Attention](https://mlsanity.com/paper/1712.05382/) (2017)
- [Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modeling](https://mlsanity.com/paper/2010.04301/) (2020)
- [ESPnet-TTS: Unified, Reproducible, and Integratable Open Source End-to-End Text-to-Speech Toolkit](https://mlsanity.com/paper/1910.10909/) (2019)
- [Robust Sequence-to-Sequence Acoustic Modeling with Stepwise Monotonic Attention for Neural TTS](https://mlsanity.com/paper/1906.00672/) (2019)
- [VQTTS: High-Fidelity Text-to-Speech Synthesis with Self-Supervised VQ Acoustic Feature](https://mlsanity.com/paper/2204.00768/) (2022)
- [ESPnet2-TTS: Extending the Edge of TTS Research](https://mlsanity.com/paper/2110.07840/) (2021)
- [DiscreTalk: Text-to-Speech as a Machine Translation Problem](https://mlsanity.com/paper/2005.05525/) (2020)
- [DenoiSpeech: Denoising Text to Speech with Frame-Level Noise Modeling](https://mlsanity.com/paper/2012.09547/) (2020)
- [AdaSpeech 3: Adaptive Text to Speech for Spontaneous Style](https://mlsanity.com/paper/2107.02530/) (2021)
- [Multi-head Monotonic Chunkwise Attention For Online Speech Recognition](https://mlsanity.com/paper/2005.00205/) (2020)
- [A study of latent monotonic attention variants](https://mlsanity.com/paper/2103.16710/) (2021)

## Built on by (most cited first, 22 of 22)

- [Taming Data and Transformers for Audio Generation](https://mlsanity.com/paper/2406.19388/) (2024)
- [SpoofCeleb: Speech Deepfake Detection and SASV In The Wild](https://mlsanity.com/paper/2409.17285/) (2024)
- [Generative Expressive Conversational Speech Synthesis](https://mlsanity.com/paper/2407.21491/) (2024)
- [TTSDS -- Text-to-Speech Distribution Score](https://mlsanity.com/paper/2407.12707/) (2024)
- [The IMS Toucan System for the Blizzard Challenge 2023](https://mlsanity.com/paper/2310.17499/) (2023)
- [Pheme: Efficient and Conversational Speech Generation](https://mlsanity.com/paper/2401.02839/) (2024)
- [Text-To-Speech Synthesis In The Wild](https://mlsanity.com/paper/2409.08711/) (2024)
- [On the Use of Self-Supervised Speech Representations in Spontaneous Speech Synthesis](https://mlsanity.com/paper/2307.05132/) (2023)
- [M3AV: A Multimodal, Multigenre, and Multipurpose Audio-Visual Academic Lecture Dataset](https://mlsanity.com/paper/2403.14168/) (2024)
- [Robust and Unbounded Length Generalization in Autoregressive Transformer-Based Text-to-Speech](https://mlsanity.com/paper/2410.22179/) (2024)
- [Prompt-Unseen-Emotion: Zero-shot Expressive Speech Synthesis with Prompt-LLM Contextual Knowledge for Mixed Emotions](https://mlsanity.com/paper/2506.02742/) (2025)
- [A Comparative Study of Self-Supervised Speech Representations in Read and Spontaneous TTS](https://mlsanity.com/paper/2303.02719/) (2023)
- [Evaluating Speech Synthesis by Training Recognizers on Synthetic Speech](https://mlsanity.com/paper/2310.00706/) (2023)
- [Expand-and-Quantize: Unsupervised Semantic Segmentation Using High-Dimensional Space and Product Quantization](https://mlsanity.com/paper/2312.07342/) (2023)
- [Continuous Autoregressive Modeling with Stochastic Monotonic Alignment for Speech Synthesis](https://mlsanity.com/paper/2502.01084/) (2025)
- [Investigating Stochastic Methods for Prosody Modeling in Speech Synthesis](https://mlsanity.com/paper/2507.00227/) (2025)
- [DC CoMix TTS: An End-to-End Expressive TTS with Discrete Code Collaborated with Mixer](https://mlsanity.com/paper/2305.19567/) (2023)
- [A Variational Framework for Improving Naturalness in Generative Spoken Language Models](https://mlsanity.com/paper/2506.14767/) (2025)
- [MultiGen: Child-Friendly Multilingual Speech Generator with LLMs](https://mlsanity.com/paper/2508.08715/) (2025)
- [Training Flow Matching Models with Reliable Labels via Self-Purification](https://mlsanity.com/paper/2509.19091/) (2025)
- [RLAIF-SPA: Structured AI Feedback for Semantic-Prosodic Alignment in Speech Synthesis](https://mlsanity.com/paper/2510.14628/) (2025)
- [MELD: Mel-Spectrogram-Based Speech Language Modeling with Discrete Latent Variables](https://mlsanity.com/paper/2605.29859/) (2026)

## Abstract (arXiv)

Recent Text-to-Speech (TTS) systems trained on reading or acted corpora have achieved near human-level naturalness. The diversity of human speech, however, often goes beyond the coverage of these corpora. We believe the ability to handle such diversity is crucial for AI systems to achieve human-level communication. Our work explores the use of more abundant real-world data for building speech synthesizers. We train TTS systems using real-world speech from YouTube and podcasts. We observe the mismatch between training and inference alignments in mel-spectrogram based autoregressive models, leading to unintelligible synthesis, and demonstrate that learned discrete codes within multiple code groups effectively resolves this issue. We introduce our MQTTS system whose architecture is designed for multiple code generation and monotonic alignment, along with the use of a clean silence prompt to improve synthesis quality. We conduct ablation analyses to identify the efficacy of our methods. We show that MQTTS outperforms existing TTS systems in several objective and subjective measures.