mlsanityData to 7 Oct 2026

A Comparative Study of Self-Supervised Speech Representations in Read and Spontaneous TTS

arXiv 2303.02719 ↗First posted 5 Mar 2023eess.AS cs.HC cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

Recent work has explored using self-supervised learning (SSL) speech representations such as wav2vec2.0 as the representation medium in standard two-stage TTS, in place of conventionally used mel-spectrograms. It is however unclear which speech SSL is the better fit for TTS, and whether or not the performance differs between read and spontaneous TTS, the later of which is arguably more challenging. This study aims at addressing these questions by testing several speech SSLs, including different layers of the same SSL, in two-stage TTS on both read and spontaneous corpora, while maintaining constant TTS model architecture and training settings. Results from listening tests show that the 9th layer of 12-layer wav2vec2.0 (ASR finetuned) outperforms other tested SSLs and mel-spectrogram, in both read and spontaneous TTS. Our work sheds light on both how speech SSL can readily improve current TTS systems, and how SSLs compare in the challenging generative task of TTS. Audio examples can be found at https://www.speech.kth.se/tts-demos/ssr_tts

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 19 of its 28 references that are in the map. First posted on arXiv 5 Mar 2023.

Builds on

19 of 28 references.

References · 19

  1. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  2. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  3. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  4. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  5. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
  6. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  7. 2021Generative Spoken Language Modeling from Raw Audiocited by 336
  8. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 293
  9. 2020Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Searchcited by 269
  10. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  11. 2021A Survey on Neural Speech Synthesiscited by 182
  12. 2020FastPitch: Parallel Text-to-speech with Pitch Predictioncited by 150
7 more
  1. 2021Generalization Ability of MOS Prediction Networkscited by 69
  2. 2021One TTS Alignment To Rule Them Allcited by 47
  3. 2022VQTTS: High-Fidelity Text-to-Speech Synthesis with Self-Supervised VQ Acoustic Featurecited by 32
  4. 2022Exploration of A Self-Supervised Speech Model: A Study on Emotional Corporacited by 31
  5. 2020Probing Acoustic Representations for Phonetic Propertiescited by 26
  6. 2023A Vector Quantized Approach for Text to Speech Synthesis on Real-World Spontaneous Speechcited by 22
  7. 2022WavThruVec: Latent speech representation as intermediate features for neural speech synthesiscited by 17

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 1.

3 papers.

  1. 2023On the Use of Self-Supervised Speech Representations in Spontaneous Speech Synthesiscited by 4
  2. 2024kNN Retrieval for Simple and Effective Zero-Shot Multi-speaker Text-to-Speechcited by 3
  3. 2023QS-TTS: Towards Semi-Supervised Text-to-Speech Synthesis via Vector-Quantized Self-Supervised Speech Representation Learningcited by 1