mlsanityData to 14 Oct 2026

QS-TTS: Towards Semi-Supervised Text-to-Speech Synthesis via Vector-Quantized Self-Supervised Speech Representation Learning

arXiv 2309.00126 ↗First posted 31 Aug 2023cs.SD cs.CL eess.AScited by 1

Not on the map. Why?

Abstract (arXiv)

This paper proposes a novel semi-supervised TTS framework, QS-TTS, to improve TTS quality with lower supervised data requirements via Vector-Quantized Self-Supervised Speech Representation Learning (VQ-S3RL) utilizing more unlabeled speech audio. This framework comprises two VQ-S3R learners: first, the principal learner aims to provide a generative Multi-Stage Multi-Codebook (MSMC) VQ-S3R via the MSMC-VQ-GAN combined with the contrastive S3RL, while decoding it back to the high-quality audio; then, the associate learner further abstracts the MSMC representation into a highly-compact VQ representation through a VQ-VAE. These two generative VQ-S3R learners provide profitable speech representations and pre-trained models for TTS, significantly improving synthesis quality with the lower requirement for supervised data. QS-TTS is evaluated comprehensively under various scenarios via subjective and objective tests in experiments. The results powerfully demonstrate the superior performance of QS-TTS, winning the highest MOS over supervised or semi-supervised baseline TTS approaches, especially in low-resource scenarios. Moreover, comparing various speech representations and transfer learning methods in TTS further validates the notable improvement of the proposed VQ-S3RL to TTS, showing the best audio quality and intelligibility metrics. The trend of slower decay in the synthesis quality of QS-TTS with decreasing supervised data further highlights its lower requirements for supervised data, indicating its great potential in low-resource scenarios.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 47 of its 63 references that are in the map. First posted on arXiv 31 Aug 2023.

Builds on

47 of 63 references.

References · 47

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2017Neural Discrete Representation Learningcited by 4,712
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  5. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  6. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  7. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  8. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,387
  9. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  10. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 804
  11. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 774
  12. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
35 more
  1. 2020Self-supervised Learning: Generative or Contrastivecited by 694
  2. 2022High Fidelity Neural Audio Compressioncited by 687
  3. 2020ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verificationcited by 628
  4. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 561
  5. 2020A Survey on Contrastive Self-supervised Learningcited by 554
  6. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 490
  7. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 340
  8. 2020Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Searchcited by 273
  9. 2020Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Imagescited by 264
  10. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 260
  11. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  12. 2018Fr'echet Audio Distance: A Metric for Evaluating Music Enhancement Algorithmscited by 225
  13. 2021WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognitioncited by 197
  14. 2023Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervisioncited by 169
  15. 2021Large-scale Self-Supervised Speech Representation Learning for Automatic Speaker Verificationcited by 78
  16. 2019Tagged Back-Translationcited by 77
  17. 2020End-to-End Adversarial Text-to-Speechcited by 77
  18. 2021UnivNet: A Neural Vocoder with Multi-Resolution Spectrogram Discriminators for High-Fidelity Waveform Generationcited by 74
  19. 2023A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AIcited by 68
  20. 2021PnG BERT: Augmented BERT on Phonemes and Graphemes for Neural TTScited by 52
  21. 2019Low Bit-Rate Speech Coding with VQ-VAE and a WaveNet Decodercited by 49
  22. 2017Listening while Speaking: Speech Chain by Deep Learningcited by 46
  23. 2020Conversational End-to-End TTS for Voice Agentcited by 37
  24. 2022VQTTS: High-Fidelity Text-to-Speech Synthesis with Self-Supervised VQ Acoustic Featurecited by 32
  25. 2018Semi-Supervised Training for Improving Data Efficiency in End-to-End Speech Synthesiscited by 30
  26. 2022SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learningcited by 26
  27. 2020DiscreTalk: Text-to-Speech as a Machine Translation Problemcited by 21
  28. 2020Simultaneous Speech-to-Speech Translation System with Neural Incremental ASR, MT, and TTScited by 12
  29. 2021Improving Streaming Transformer Based ASR Under a Framework of Self-supervised Learningcited by 12
  30. 2022Transfer Learning Framework for Low-Resource Text-to-Speech using a Large-Scale Unlabeled Speech Corpuscited by 10
  31. 2022Joint Training of Speech Enhancement and Self-supervised Model for Noise-robust ASRcited by 9
  32. 2022A Multi-Stage Multi-Codebook VQ-VAE Approach to High-Performance Neural TTScited by 9
  33. 2021VAENAR-TTS: Variational Auto-Encoder based Non-AutoRegressive Text-to-Speech Synthesiscited by 7
  34. 2020Semi-supervised Learning for Multi-speaker Text-to-speech Synthesis Using Discrete Speech Representationcited by 4
  35. 2023A Comparative Study of Self-Supervised Speech Representations in Read and Spontaneous TTScited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Iterative Self-Learning for Expressive Text-to-Speech Synthesiscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023An Experimental Study: Assessing the Combined Framework of WavLM and BEST-RQ for Text-to-Speech Synthesis
  2. 2025Quantize More, Lose Less: Autoregressive Generation from Residually Quantized Speech Representations
  3. 2022DelightfulTTS 2: End-to-End Speech Synthesis with Adversarial Vector-Quantized Auto-Encoders
  4. 2023A Vector Quantized Approach for Text to Speech Synthesis on Real-World Spontaneous Speech
  5. 2021Preliminary study on using vector quantization latent spaces for TTS/VC systems with consistent performance
  6. 2026Qwen-Audio-3.0-TTS: Freely Controllable and Highly Robust Speech Synthesis with Multi-Stage Training Paradigm
  7. 2023High-Fidelity Speech Synthesis with Minimal Supervision: All Using Diffusion Models
  8. 2021Wav2vec-C: A Self-supervised Model for Speech Representation Learning