mlsanityData to 14 Oct 2026

StyleSpeech: Self-supervised Style Enhancing with VQ-VAE-based Pre-training for Expressive Audiobook Speech Synthesis

arXiv 2312.12181 ↗First posted 19 Dec 2023cs.SD cs.AI eess.AScited by 4

Not on the map. Why?

Abstract (arXiv)

The expressive quality of synthesized speech for audiobooks is limited by generalized model architecture and unbalanced style distribution in the training data. To address these issues, in this paper, we propose a self-supervised style enhancing method with VQ-VAE-based pre-training for expressive audiobook speech synthesis. Firstly, a text style encoder is pre-trained with a large amount of unlabeled text-only data. Secondly, a spectrogram style extractor based on VQ-VAE is pre-trained in a self-supervised manner, with plenty of audio data that covers complex style variations. Then a novel architecture with two encoder-decoder paths is specially designed to model the pronunciation and high-level style expressiveness respectively, with the guidance of the style extractor. Both objective and subjective evaluations demonstrate that our proposed method can effectively improve the naturalness and expressiveness of the synthesized speech in audiobook synthesis especially for the role and out-of-domain scenarios.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 18 of its 25 references that are in the map. First posted on arXiv 19 Dec 2023.

Builds on

18 of 25 references.

References · 18

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2017Neural Discrete Representation Learningcited by 4,712
  5. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  6. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 804
  7. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 717
  8. 2018Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesiscited by 259
  9. 2018Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotroncited by 173
  10. 2021Supporting Clustering with Contrastive Learningcited by 90
  11. 2018Predicting Expressive Speaking Style From Text In End-To-End Speech Synthesiscited by 32
  12. 2020Improving Prosody Modelling with Cross-Utterance BERT Embeddings for End-to-end Speech Synthesiscited by 27
6 more
  1. 2020Learning and Evaluating Emotion Lexicons for 91 Languagescited by 13
  2. 2020Learning Disentangled Phone and Speaker Representations in a Semi-Supervised VQ-VAE Paradigmcited by 12
  3. 2022Self-supervised Context-aware Style Representation for Expressive Speech Synthesiscited by 10
  4. 2022A Character-level Span-based Model for Mandarin Prosodic Structure Predictioncited by 4
  5. 2022Towards Multi-Scale Speaking Style Modelling with Hierarchical Context Information for Mandarin Speech Synthesiscited by 4
  6. 2020Improved Prosody from Learned F0 Codebook Representations for VQ-VAE Speech Waveform Reconstructioncited by 3

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 2.

4 papers.

  1. 2024Style Mixture of Experts for Expressive Text-To-Speech Synthesiscited by 6
  2. 2025DiffDSR: Dysarthric Speech Reconstruction Using Latent Diffusion Modelcited by 5
  3. 2024CoLM-DSR: Leveraging Neural Codec Language Modeling for Multi-Modal Dysarthric Speech Reconstructioncited by 4
  4. 2025See the Speaker: Crafting High-Resolution Talking Faces from Speech with Prior Guidance and Region Refinementcited by 0