# Pretraining Techniques for Sequence-to-Sequence Voice Conversion

arXiv: https://arxiv.org/abs/2008.03088 · HTML: https://mlsanity.com/paper/2008.03088/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Aug 2020.

## Built on by (most cited first, 13 of 13)

- [SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing](https://mlsanity.com/paper/2110.07205/) (2021)
- [Emotion Intensity and its Control for Emotional Voice Conversion](https://mlsanity.com/paper/2201.03967/) (2022)
- [The Singing Voice Conversion Challenge 2023](https://mlsanity.com/paper/2306.14422/) (2023)
- [Towards Identity Preserving Normal to Dysarthric Voice Conversion](https://mlsanity.com/paper/2110.08213/) (2021)
- [A Preliminary Study of a Two-Stage Paradigm for Preserving Speaker Identity in Dysarthric Voice Conversion](https://mlsanity.com/paper/2106.01415/) (2021)
- [Evaluating Methods for Ground-Truth-Free Foreign Accent Conversion](https://mlsanity.com/paper/2309.02133/) (2023)
- [On Prosody Modeling for ASR+TTS based Voice Conversion](https://mlsanity.com/paper/2107.09477/) (2021)
- [VQalAttent: a Transparent Speech Generation Pipeline based on Transformer-learned VQ-VAE Latent Space](https://mlsanity.com/paper/2411.14642/) (2024)
- [A Pilot Study of Applying Sequence-to-Sequence Voice Conversion to Evaluate the Intelligibility of L2 Speech Using a Native Speaker's Shadowings](https://mlsanity.com/paper/2410.02239/) (2024)
- [Hyperpruning: Efficient Search through Pruned Variants of Recurrent Neural Networks Leveraging Lyapunov Spectrum](https://mlsanity.com/paper/2506.07975/) (2025)
- [Preliminary study on using vector quantization latent spaces for TTS/VC systems with consistent performance](https://mlsanity.com/paper/2106.13479/) (2021)
- [Time Alignment using Lip Images for Frame-based Electrolaryngeal Voice Conversion](https://mlsanity.com/paper/2109.03551/) (2021)
- [Parallel and Limited Data Voice Conversion Using Stochastic Variational Deep Kernel Learning](https://mlsanity.com/paper/2309.04420/) (2023)

## Abstract (arXiv)

Sequence-to-sequence (seq2seq) voice conversion (VC) models are attractive owing to their ability to convert prosody. Nonetheless, without sufficient data, seq2seq VC models can suffer from unstable training and mispronunciation problems in the converted speech, thus far from practical. To tackle these shortcomings, we propose to transfer knowledge from other speech processing tasks where large-scale corpora are easily available, typically text-to-speech (TTS) and automatic speech recognition (ASR). We argue that VC models initialized with such pretrained ASR or TTS model parameters can generate effective hidden representations for high-fidelity, highly intelligible converted speech. We apply such techniques to recurrent neural network (RNN)-based and Transformer based models, and through systematical experiments, we demonstrate the effectiveness of the pretraining scheme and the superiority of Transformer based models over RNN-based models in terms of intelligibility, naturalness, and similarity.