mlsanityData to 7 Oct 2026

Zero-shot text-to-speech synthesis conditioned using self-supervised speech representation model

arXiv 2304.11976 ↗First posted 24 Apr 2023cs.SD cs.LG eess.AScited by 10

Not on the map. Why?

Abstract (arXiv)

This paper proposes a zero-shot text-to-speech (TTS) conditioned by a self-supervised speech-representation model acquired through self-supervised learning (SSL). Conventional methods with embedding vectors from x-vector or global style tokens still have a gap in reproducing the speaker characteristics of unseen speakers. A novel point of the proposed method is the direct use of the SSL model to obtain embedding vectors from speech representations trained with a large amount of data. We also introduce the separate conditioning of acoustic features and a phoneme duration predictor to obtain the disentangled embeddings between rhythm-based speaker characteristics and acoustic-feature-based ones. The disentangled embeddings will enable us to achieve better reproduction performance for unseen speakers and rhythm transfer conditioned by different speeches. Objective and subjective evaluations showed that the proposed method can synthesize speech with improved similarity and achieve speech-rhythm transfer.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 25 of its 36 references that are in the map. First posted on arXiv 24 Apr 2023.

Builds on

25 of 36 references.

References · 25

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  4. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  5. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  6. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  7. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
  8. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  9. 2018Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesiscited by 257
  10. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  11. 2021Direct speech-to-speech translation with discrete unitscited by 148
  12. 2020In defence of metric learning for speaker recognitioncited by 123
13 more
  1. 2019Zero-Shot Multi-Speaker Text-To-Speech with State-of-the-art Neural Speaker Embeddingscited by 73
  2. 2015End-to-End Text-Dependent Speaker Verificationcited by 49
  3. 2018Robust and fine-grained prosody control of end-to-end speech synthesiscited by 48
  4. 2019Mellotron: Multispeaker expressive voice synthesis by conditioning on rhythm, pitch and global style tokenscited by 44
  5. 2020Fully-hierarchical fine-grained prosody modeling for interpretable speech synthesiscited by 41
  6. 2020FragmentVC: Any-to-Any Voice Conversion by End-to-End Extracting and Fusing Fine-Grained Voice Fragments With Attentioncited by 37
  7. 2021Investigating on Incorporating Pretrained and Learnable Speaker Representations for Multi-Speaker Multi-Style Text-to-Speechcited by 37
  8. 2021Universal Paralinguistic Speech Representations Using Self-Supervised Conformerscited by 32
  9. 2022Why does Self-Supervised Learning for Speech Recognition Benefit Speaker Recognition?cited by 27
  10. 2022GenerSpeech: Towards Style Transfer for Generalizable Out-Of-Domain Text-to-Speechcited by 21
  11. 2022Streaming Target-Speaker ASR with Neural Transducercited by 8
  12. 2022Learning to Enhance or Not: Neural Network-Based Switching of Enhanced and Observed Signals for Overlapping Speech Recognitioncited by 7
  13. 2021Should We Always Separate?: Switching Between Enhanced and Observed Signals for Overlapping Speech Recognitioncited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 3; 2025: 3; 2026: 2.

10 papers.

  1. 2023StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Modelscited by 148
  2. 2023StyleCap: Automatic Speaking-Style Captioning from Speech Based on Speech and Language Self-supervised Learning Modelscited by 7
  3. 2024Noise-robust zero-shot text-to-speech synthesis conditioned on self-supervised speech-representation model with adapterscited by 4
  4. 2024Description-based Controllable Text-to-Speech with Cross-Lingual Voice Controlcited by 4
  5. 2025Voice Cloning: Comprehensive Surveycited by 4
  6. 2025Voice Impression Control in Zero-Shot TTScited by 3
  7. 2024Lightweight Zero-shot Text-to-Speech with Mixture of Adapterscited by 2
  8. 2025Multi-interaction TTS toward professional recording reproductioncited by 2
  9. 2026Investigation for Relative Voice Impression Estimationcited by 1
  10. 2026Scalable Direction-Following TTS via Voice Impression-Guided Pseudo Triplet Constructioncited by 0