mlsanityData to 14 Oct 2026

On the Use of Self-Supervised Speech Representations in Spontaneous Speech Synthesis

arXiv 2307.05132 ↗First posted 11 Jul 2023eess.AS cs.HC cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Self-supervised learning (SSL) speech representations learned from large amounts of diverse, mixed-quality speech data without transcriptions are gaining ground in many speech technology applications. Prior work has shown that SSL is an effective intermediate representation in two-stage text-to-speech (TTS) for both read and spontaneous speech. However, it is still not clear which SSL and which layer from each SSL model is most suited for spontaneous TTS. We address this shortcoming by extending the scope of comparison for SSL in spontaneous TTS to 6 different SSLs and 3 layers within each SSL. Furthermore, SSL has also shown potential in predicting the mean opinion scores (MOS) of synthesized speech, but this has only been done in read-speech MOS prediction. We extend an SSL-based MOS prediction framework previously developed for scoring read speech synthesis and evaluate its performance on synthesized spontaneous speech. All experiments are conducted twice on two different spontaneous corpora in order to find generalizable trends. Overall, we present comprehensive experimental results on the use of SSL in spontaneous TTS and MOS prediction to further quantify and understand how SSL can be used in spontaneous TTS. Audios samples: https://www.speech.kth.se/tts-demos/sp_ssl_tts

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 25 of its 35 references that are in the map. First posted on arXiv 11 Jul 2023.

Builds on

25 of 35 references.

References · 25

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  3. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  4. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  5. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  6. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  7. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 399
  8. 2023A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPTcited by 368
  9. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 298
  10. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 260
  11. 2021A Survey on Neural Speech Synthesiscited by 184
  12. 2023Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervisioncited by 169
13 more
  1. 2020FastPitch: Parallel Text-to-speech with Pitch Predictioncited by 151
  2. 2021Generalization Ability of MOS Prediction Networkscited by 69
  3. 2022The VoiceMOS Challenge 2022cited by 57
  4. 2021One TTS Alignment To Rule Them Allcited by 47
  5. 2022VQTTS: High-Fidelity Text-to-Speech Synthesis with Self-Supervised VQ Acoustic Featurecited by 32
  6. 2022Exploration of A Self-Supervised Speech Model: A Study on Emotional Corporacited by 31
  7. 2023A Vector Quantized Approach for Text to Speech Synthesis on Real-World Spontaneous Speechcited by 22
  8. 2021Utilizing Self-supervised Representations for MOS Predictioncited by 20
  9. 2022WavThruVec: Latent speech representation as intermediate features for neural speech synthesiscited by 17
  10. 2022ConferencingSpeech 2022 Challenge: Non-intrusive Objective Speech Quality Assessment (NISQA) Challenge for Online Conferencing Applicationscited by 15
  11. 2022Pre-trained Speech Representations as Feature Extractors for Speech Quality Assessment in Online Conferencing Applicationscited by 7
  12. 2022Exploring the influence of fine-tuning data on wav2vec 2.0 model for blind speech quality predictioncited by 5
  13. 2023A Comparative Study of Self-Supervised Speech Representations in Read and Spontaneous TTScited by 3

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 2; 2025: 1.

4 papers.

  1. 2023Unified speech and gesture synthesis using flow matchingcited by 6
  2. 2025Segmentation-Variant Codebooks for Preservation of Paralinguistic and Prosodic Informationcited by 4
  3. 2024kNN Retrieval for Simple and Effective Zero-Shot Multi-speaker Text-to-Speechcited by 3
  4. 2024Text-to-Speech for Unseen Speakers via Low-Complexity Discrete Unit-Based Frame Selectioncited by 2