# Transfer Learning from Speech Synthesis to Voice Conversion with Non-Parallel Training Data

arXiv: https://arxiv.org/abs/2009.14399 · HTML: https://mlsanity.com/paper/2009.14399/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Sep 2020.

## Built on by (most cited first, 17 of 17)

- [Emotional Voice Conversion: Theory, Databases and ESD](https://mlsanity.com/paper/2105.14762/) (2021)
- [FreeVC: Towards High-Quality Text-Free One-Shot Voice Conversion](https://mlsanity.com/paper/2210.15418/) (2022)
- [Robust Disentangled Variational Speech Representation Learning for Zero-shot Voice Conversion](https://mlsanity.com/paper/2203.16705/) (2022)
- [Speech Synthesis with Mixed Emotions](https://mlsanity.com/paper/2208.05890/) (2022)
- [Towards Improved Zero-shot Voice Conversion with Conditional DSVAE](https://mlsanity.com/paper/2205.05227/) (2022)
- [VQ-CTAP: Cross-Modal Fine-Grained Sequence Representation Learning for Speech Processing](https://mlsanity.com/paper/2408.05758/) (2024)
- [AV-TranSpeech: Audio-Visual Robust Speech-to-Speech Translation](https://mlsanity.com/paper/2305.15403/) (2023)
- [EMA2S: An End-to-End Multimodal Articulatory-to-Speech System](https://mlsanity.com/paper/2102.03786/) (2021)
- [NoiseVC: Towards High Quality Zero-Shot Voice Conversion](https://mlsanity.com/paper/2104.06074/) (2021)
- [Residual Information in Deep Speaker Embedding Architectures](https://mlsanity.com/paper/2302.02742/) (2023)
- [Robust Channel Learning for Large-Scale Radio Speaker Verification](https://mlsanity.com/paper/2406.10956/) (2024)
- [Synth4Kws: Synthesized Speech for User Defined Keyword Spotting in Low Resource Environments](https://mlsanity.com/paper/2407.16840/) (2024)
- [Boosting Star-GANs for Voice Conversion with Contrastive Discriminator](https://mlsanity.com/paper/2209.10088/) (2022)
- [Preliminary study on using vector quantization latent spaces for TTS/VC systems with consistent performance](https://mlsanity.com/paper/2106.13479/) (2021)
- [Parallel and Limited Data Voice Conversion Using Stochastic Variational Deep Kernel Learning](https://mlsanity.com/paper/2309.04420/) (2023)
- [BnTTS: Few-Shot Speaker Adaptation in Low-Resource Setting](https://mlsanity.com/paper/2502.05729/) (2025)
- [From A to B to A: Palindromic Zero-Shot Voice Conversion with Non-Parallel Data](https://mlsanity.com/paper/2606.08843/) (2026)

## Abstract (arXiv)

This paper presents a novel framework to build a voice conversion (VC) system by learning from a text-to-speech (TTS) synthesis system, that is called TTS-VC transfer learning. We first develop a multi-speaker speech synthesis system with sequence-to-sequence encoder-decoder architecture, where the encoder extracts robust linguistic representations of text, and the decoder, conditioned on target speaker embedding, takes the context vectors and the attention recurrent network cell output to generate target acoustic features. We take advantage of the fact that TTS system maps input text to speaker independent context vectors, and reuse such a mapping to supervise the training of latent representations of an encoder-decoder voice conversion system. In the voice conversion system, the encoder takes speech instead of text as input, while the decoder is functionally similar to TTS decoder. As we condition the decoder on speaker embedding, the system can be trained on non-parallel data for any-to-any voice conversion. During voice conversion training, we present both text and speech to speech synthesis and voice conversion networks respectively. At run-time, the voice conversion network uses its own encoder-decoder architecture. Experiments show that the proposed approach outperforms two competitive voice conversion baselines consistently, namely phonetic posteriorgram and variational autoencoder methods, in terms of speech quality, naturalness, and speaker similarity.