mlsanityData to 7 Oct 2026

A Vector Quantized Approach for Text to Speech Synthesis on Real-World Spontaneous Speech

arXiv 2302.04215 ↗First posted 8 Feb 2023eess.AS cs.AI cs.LGcited by 22

Not on the map. Why?

Abstract (arXiv)

Recent Text-to-Speech (TTS) systems trained on reading or acted corpora have achieved near human-level naturalness. The diversity of human speech, however, often goes beyond the coverage of these corpora. We believe the ability to handle such diversity is crucial for AI systems to achieve human-level communication. Our work explores the use of more abundant real-world data for building speech synthesizers. We train TTS systems using real-world speech from YouTube and podcasts. We observe the mismatch between training and inference alignments in mel-spectrogram based autoregressive models, leading to unintelligible synthesis, and demonstrate that learned discrete codes within multiple code groups effectively resolves this issue. We introduce our MQTTS system whose architecture is designed for multiple code generation and monotonic alignment, along with the use of a clean silence prompt to improve synthesis quality. We conduct ablation analyses to identify the efficacy of our methods. We show that MQTTS outperforms existing TTS systems in several objective and subjective measures.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 29 of its 39 references that are in the map. First posted on arXiv 8 Feb 2023.

Builds on

29 of 39 references.

References · 29

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  4. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  5. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  6. 2018Group Normalizationcited by 1,514
  7. 2018Hierarchical Neural Story Generationcited by 1,182
  8. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  9. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  10. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  11. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 555
  12. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 554
17 more
  1. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 391
  2. 2021Generative Spoken Language Modeling from Raw Audiocited by 336
  3. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
  4. 2020Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Searchcited by 269
  5. 2018Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesiscited by 257
  6. 2022Dawn of the transformer era in speech emotion recognition: closing the valence gapcited by 155
  7. 2017Monotonic Chunkwise Attentioncited by 82
  8. 2020Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modelingcited by 65
  9. 2019ESPnet-TTS: Unified, Reproducible, and Integratable Open Source End-to-End Text-to-Speech Toolkitcited by 60
  10. 2019Robust Sequence-to-Sequence Acoustic Modeling with Stepwise Monotonic Attention for Neural TTScited by 33
  11. 2022VQTTS: High-Fidelity Text-to-Speech Synthesis with Self-Supervised VQ Acoustic Featurecited by 32
  12. 2021ESPnet2-TTS: Extending the Edge of TTS Researchcited by 22
  13. 2020DiscreTalk: Text-to-Speech as a Machine Translation Problemcited by 21
  14. 2020DenoiSpeech: Denoising Text to Speech with Frame-Level Noise Modelingcited by 13
  15. 2021AdaSpeech 3: Adaptive Text to Speech for Spontaneous Stylecited by 11
  16. 2020Multi-head Monotonic Chunkwise Attention For Online Speech Recognitioncited by 5
  17. 2021A study of latent monotonic attention variantscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 8; 2025: 7; 2026: 1.

22 papers.

Sort:

By area:

  1. 2024Taming Data and Transformers for Audio Generationcited by 28
  2. 2024SpoofCeleb: Speech Deepfake Detection and SASV In The Wildcited by 22
  3. 2024Generative Expressive Conversational Speech Synthesiscited by 16
  4. 2024TTSDS -- Text-to-Speech Distribution Scorecited by 14
  5. 2023The IMS Toucan System for the Blizzard Challenge 2023cited by 6
  6. 2024Pheme: Efficient and Conversational Speech Generationcited by 6
  7. 2024Text-To-Speech Synthesis In The Wildcited by 6
  8. 2023On the Use of Self-Supervised Speech Representations in Spontaneous Speech Synthesiscited by 4
  9. 2024Robust and Unbounded Length Generalization in Autoregressive Transformer-Based Text-to-Speechcited by 4
  10. 2025Prompt-Unseen-Emotion: Zero-shot Expressive Speech Synthesis with Prompt-LLM Contextual Knowledge for Mixed Emotionscited by 4
  11. 2023A Comparative Study of Self-Supervised Speech Representations in Read and Spontaneous TTScited by 3
  12. 2023Evaluating Speech Synthesis by Training Recognizers on Synthetic Speechcited by 3
10 more
  1. 2023Expand-and-Quantize: Unsupervised Semantic Segmentation Using High-Dimensional Space and Product Quantizationcited by 3
  2. 2024M3AV: A Multimodal, Multigenre, and Multipurpose Audio-Visual Academic Lecture Datasetcited by 3
  3. 2025Continuous Autoregressive Modeling with Stochastic Monotonic Alignment for Speech Synthesiscited by 2
  4. 2025Investigating Stochastic Methods for Prosody Modeling in Speech Synthesiscited by 2
  5. 2023DC CoMix TTS: An End-to-End Expressive TTS with Discrete Code Collaborated with Mixercited by 1
  6. 2025A Variational Framework for Improving Naturalness in Generative Spoken Language Modelscited by 1
  7. 2025MultiGen: Child-Friendly Multilingual Speech Generator with LLMscited by 1
  8. 2025Training Flow Matching Models with Reliable Labels via Self-Purificationcited by 1
  9. 2025RLAIF-SPA: Structured AI Feedback for Semantic-Prosodic Alignment in Speech Synthesiscited by 1
  10. 2026MELD: Mel-Spectrogram-Based Speech Language Modeling with Discrete Latent Variablescited by 0