# Cotatron: Transcription-Guided Speech Encoder for Any-to-Many Voice Conversion without Parallel Data

arXiv: https://arxiv.org/abs/2005.03295 · HTML: https://mlsanity.com/paper/2005.03295/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 May 2020.

## Built on by (most cited first, 14 of 14)

- [Deepfakes Generation and Detection: State-of-the-art, open challenges, countermeasures, and way forward](https://mlsanity.com/paper/2103.00484/) (2021)
- [Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised Representations](https://mlsanity.com/paper/2110.14513/) (2021)
- [FreeVC: Towards High-Quality Text-Free One-Shot Voice Conversion](https://mlsanity.com/paper/2210.15418/) (2022)
- [StarGANv2-VC: A Diverse, Unsupervised, Non-parallel Framework for Natural-Sounding Voice Conversion](https://mlsanity.com/paper/2107.10394/) (2021)
- [StyleTTS: A Style-Based Generative Model for Natural and Diverse Text-to-Speech Synthesis](https://mlsanity.com/paper/2205.15439/) (2022)
- [Assem-VC: Realistic Voice Conversion by Assembling Modern Speech Synthesis Techniques](https://mlsanity.com/paper/2104.00931/) (2021)
- [RT-VC: Real-Time Zero-Shot Voice Conversion with Speech Articulatory Coding](https://mlsanity.com/paper/2506.10289/) (2025)
- [An Adaptive Learning based Generative Adversarial Network for One-To-One Voice Conversion](https://mlsanity.com/paper/2104.12159/) (2021)
- [Speak Like a Dog: Human to Non-human creature Voice Conversion](https://mlsanity.com/paper/2206.04780/) (2022)
- [Learning Robust Latent Representations for Controllable Speech Synthesis](https://mlsanity.com/paper/2105.04458/) (2021)
- [GlowVC: Mel-spectrogram space disentangling model for language-independent text-free voice conversion](https://mlsanity.com/paper/2207.01454/) (2022)
- [Parallel and Limited Data Voice Conversion Using Stochastic Variational Deep Kernel Learning](https://mlsanity.com/paper/2309.04420/) (2023)
- [From A to B to A: Palindromic Zero-Shot Voice Conversion with Non-Parallel Data](https://mlsanity.com/paper/2606.08843/) (2026)
- [Linguistically Augmented Audio Speech Data (LinguAS)](https://mlsanity.com/paper/2606.10246/) (2026)

## Abstract (arXiv)

We propose Cotatron, a transcription-guided speech encoder for speaker-independent linguistic representation. Cotatron is based on the multispeaker TTS architecture and can be trained with conventional TTS datasets. We train a voice conversion system to reconstruct speech with Cotatron features, which is similar to the previous methods based on Phonetic Posteriorgram (PPG). By training and evaluating our system with 108 speakers from the VCTK dataset, we outperform the previous method in terms of both naturalness and speaker similarity. Our system can also convert speech from speakers that are unseen during training, and utilize ASR to automate the transcription with minimal reduction of the performance. Audio samples are available at https://mindslab-ai.github.io/cotatron, and the code with a pre-trained model will be made available soon.