# Voice Transformer Network: Sequence-to-Sequence Voice Conversion Using Transformer with Text-to-Speech Pretraining

arXiv: https://arxiv.org/abs/1912.06813 · HTML: https://mlsanity.com/paper/1912.06813/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Dec 2019.

## Built on by (most cited first, 29 of 29)

- [Emotional Voice Conversion: Theory, Databases and ESD](https://mlsanity.com/paper/2105.14762/) (2021)
- [Deepfakes Generation and Detection: State-of-the-art, open challenges, countermeasures, and way forward](https://mlsanity.com/paper/2103.00484/) (2021)
- [A Text-guided Protein Design Framework](https://mlsanity.com/paper/2302.04611/) (2023)
- [StarGANv2-VC: A Diverse, Unsupervised, Non-parallel Framework for Natural-Sounding Voice Conversion](https://mlsanity.com/paper/2107.10394/) (2021)
- [Transformers in Speech Processing: A Survey](https://mlsanity.com/paper/2303.11607/) (2023)
- [Speech Synthesis with Mixed Emotions](https://mlsanity.com/paper/2208.05890/) (2022)
- [The Singing Voice Conversion Challenge 2023](https://mlsanity.com/paper/2306.14422/) (2023)
- [i-Code: An Integrative and Composable Multimodal Learning Framework](https://mlsanity.com/paper/2205.01818/) (2022)
- [ESPnet2-TTS: Extending the Edge of TTS Research](https://mlsanity.com/paper/2110.07840/) (2021)
- [SpeechNet: A Universal Modularized Model for Speech Processing Tasks](https://mlsanity.com/paper/2105.03070/) (2021)
- [The Age of Synthetic Realities: Challenges and Opportunities](https://mlsanity.com/paper/2306.11503/) (2023)
- [Assem-VC: Realistic Voice Conversion by Assembling Modern Speech Synthesis Techniques](https://mlsanity.com/paper/2104.00931/) (2021)
- [Towards Identity Preserving Normal to Dysarthric Voice Conversion](https://mlsanity.com/paper/2110.08213/) (2021)
- [Multi-View Self-Attention Based Transformer for Speaker Recognition](https://mlsanity.com/paper/2110.05036/) (2021)
- [Non-autoregressive sequence-to-sequence voice conversion](https://mlsanity.com/paper/2104.06793/) (2021)
- [A Preliminary Study of a Two-Stage Paradigm for Preserving Speaker Identity in Dysarthric Voice Conversion](https://mlsanity.com/paper/2106.01415/) (2021)
- [An Overview & Analysis of Sequence-to-Sequence Emotional Voice Conversion](https://mlsanity.com/paper/2203.15873/) (2022)
- [EMA2S: An End-to-End Multimodal Articulatory-to-Speech System](https://mlsanity.com/paper/2102.03786/) (2021)
- [Evaluating Methods for Ground-Truth-Free Foreign Accent Conversion](https://mlsanity.com/paper/2309.02133/) (2023)
- [On Prosody Modeling for ASR+TTS based Voice Conversion](https://mlsanity.com/paper/2107.09477/) (2021)
- [Disentangling segmental and prosodic factors to non-native speech comprehensibility](https://mlsanity.com/paper/2408.10997/) (2024)
- [Axial Residual Networks for CycleGAN-based Voice Conversion](https://mlsanity.com/paper/2102.08075/) (2021)
- [GlowVC: Mel-spectrogram space disentangling model for language-independent text-free voice conversion](https://mlsanity.com/paper/2207.01454/) (2022)
- [Multilingual Prosody Transfer: Comparing Supervised & Transfer Learning](https://mlsanity.com/paper/2406.00022/) (2024)
- [Generative Adversarial Network based Voice Conversion: Techniques, Challenges, and Recent Advancements](https://mlsanity.com/paper/2504.19197/) (2025)
- [Enhancing Zero-Shot Many to Many Voice Conversion with Self-Attention VAE](https://mlsanity.com/paper/2203.16037/) (2022)
- [A Pilot Study of Applying Sequence-to-Sequence Voice Conversion to Evaluate the Intelligibility of L2 Speech Using a Native Speaker's Shadowings](https://mlsanity.com/paper/2410.02239/) (2024)
- [Time Alignment using Lip Images for Frame-based Electrolaryngeal Voice Conversion](https://mlsanity.com/paper/2109.03551/) (2021)
- [Parallel and Limited Data Voice Conversion Using Stochastic Variational Deep Kernel Learning](https://mlsanity.com/paper/2309.04420/) (2023)

## Abstract (arXiv)

We introduce a novel sequence-to-sequence (seq2seq) voice conversion (VC) model based on the Transformer architecture with text-to-speech (TTS) pretraining. Seq2seq VC models are attractive owing to their ability to convert prosody. While seq2seq models based on recurrent neural networks (RNNs) and convolutional neural networks (CNNs) have been successfully applied to VC, the use of the Transformer network, which has shown promising results in various speech processing tasks, has not yet been investigated. Nonetheless, their data-hungry property and the mispronunciation of converted speech make seq2seq models far from practical. To this end, we propose a simple yet effective pretraining technique to transfer knowledge from learned TTS models, which benefit from large-scale, easily accessible TTS corpora. VC models initialized with such pretrained model parameters are able to generate effective hidden representations for high-fidelity, highly intelligible converted speech. Experimental results show that such a pretraining scheme can facilitate data-efficient training and outperform an RNN-based seq2seq VC model in terms of intelligibility, naturalness, and similarity.