mlsanityData to 7 Oct 2026

UnifySpeech: A Unified Framework for Zero-shot Text-to-Speech and Voice Conversion

arXiv 2301.03801 ↗First posted 10 Jan 2023cs.SD cs.CL cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

Text-to-speech (TTS) and voice conversion (VC) are two different tasks both aiming at generating high quality speaking voice according to different input modality. Due to their similarity, this paper proposes UnifySpeech, which brings TTS and VC into a unified framework for the first time. The model is based on the assumption that speech can be decoupled into three independent components: content information, speaker information, prosody information. Both TTS and VC can be regarded as mining these three parts of information from the input and completing the reconstruction of speech. For TTS, the speech content information is derived from the text, while in VC it's derived from the source speech, so all the remaining units are shared except for the speech content extraction module in the two tasks. We applied vector quantization and domain constrain to bridge the gap between the content domains of TTS and VC. Objective and subjective evaluation shows that by combining the two task, TTS obtains better speaker modeling ability while VC gets hold of impressive speech content decoupling capability.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 18 of its 24 references that are in the map. First posted on arXiv 10 Jan 2023.

Builds on

18 of 24 references.

References · 18

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2017Neural Discrete Representation Learningcited by 4,650
  4. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  5. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
  6. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  7. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 555
  8. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 487
  9. 2020Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Searchcited by 269
  10. 2018Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesiscited by 257
  11. 2016Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learningcited by 254
  12. 2020An Overview of Voice Conversion and its Challenges: From Statistical Modeling to Deep Learningcited by 111
6 more
  1. 2021Meta-StyleSpeech : Multi-Speaker Adaptive Text-to-Speech Generationcited by 93
  2. 2019Location-Relative Attention Mechanisms For Robust Long-Form Speech Synthesiscited by 44
  3. 2021AdaSpeech 2: Adaptive Text to Speech with Untranscribed Datacited by 17
  4. 2022Glow-WaveGAN 2: High-quality Zero-shot Text-to-speech Synthesis and Any-to-any Voice Conversioncited by 5
  5. 2021Zero-Shot Text-to-Speech for Text-Based Insertion in Audio Narrationcited by 4
  6. 2022Zero-Shot Long-Form Voice Cloning with Dynamic Convolution Attentioncited by 2

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 0; 2025: 1.

3 papers.

  1. 2023SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Modelscited by 85
  2. 2023Make-A-Voice: Unified Voice Synthesis With Discrete Representationcited by 27
  3. 2025Voice Cloning: Comprehensive Surveycited by 4