# Voice Conversion from Non-parallel Corpora Using Variational Auto-encoder

arXiv: https://arxiv.org/abs/1610.04019 · HTML: https://mlsanity.com/paper/1610.04019/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Oct 2016.

## Built on by (most cited first, 30 of 45)

- [Emotional Voice Conversion: Theory, Databases and ESD](https://mlsanity.com/paper/2105.14762/) (2021)
- [ContentVec: An Improved Self-Supervised Speech Representation by Disentangling Speakers](https://mlsanity.com/paper/2204.09224/) (2022)
- [A Large-Scale Evaluation of Speech Foundation Models](https://mlsanity.com/paper/2404.09385/) (2024)
- [An Overview of Affective Speech Synthesis and Conversion in the Deep Learning Era](https://mlsanity.com/paper/2210.03538/) (2022)
- [SpeechSplit 2.0: Unsupervised speech disentanglement for voice conversion Without tuning autoencoder Bottlenecks](https://mlsanity.com/paper/2203.14156/) (2022)
- [SEF-VC: Speaker Embedding Free Zero-Shot Voice Conversion with Cross Attention](https://mlsanity.com/paper/2312.08676/) (2023)
- [MaskCycleGAN-VC: Learning Non-parallel Voice Conversion with Filling in Frames](https://mlsanity.com/paper/2102.12841/) (2021)
- [Adversarially learning disentangled speech representations for robust multi-factor voice conversion](https://mlsanity.com/paper/2102.00184/) (2021)
- [NVC-Net: End-to-End Adversarial Voice Conversion](https://mlsanity.com/paper/2106.00992/) (2021)
- [A Comparative Study of Self-supervised Speech Representation Based Voice Conversion](https://mlsanity.com/paper/2207.04356/) (2022)
- [SLIM: Style-Linguistics Mismatch Model for Generalized Audio Deepfake Detection](https://mlsanity.com/paper/2407.18517/) (2024)
- [Many-to-Many Voice Conversion based Feature Disentanglement using Variational Autoencoder](https://mlsanity.com/paper/2107.06642/) (2021)
- [An Improved StarGAN for Emotional Voice Conversion: Enhancing Voice Quality and Data Augmentation](https://mlsanity.com/paper/2107.08361/) (2021)
- [Training Robust Zero-Shot Voice Conversion Models with Self-supervised Features](https://mlsanity.com/paper/2112.04424/) (2021)
- [FastVoiceGrad: One-step Diffusion-Based Voice Conversion with Adversarial Conditional Diffusion Distillation](https://mlsanity.com/paper/2409.02245/) (2024)
- [A Preliminary Study of a Two-Stage Paradigm for Preserving Speaker Identity in Dysarthric Voice Conversion](https://mlsanity.com/paper/2106.01415/) (2021)
- [Global Rhythm Style Transfer Without Text Transcriptions](https://mlsanity.com/paper/2106.08519/) (2021)
- [Towards General-Purpose Text-Instruction-Guided Voice Conversion](https://mlsanity.com/paper/2309.14324/) (2023)
- [TrustEMG-Net: Using Representation-Masking Transformer with U-Net for Surface Electromyography Enhancement](https://mlsanity.com/paper/2410.03843/) (2024)
- [Learning robust speech representation with an articulatory-regularized variational autoencoder](https://mlsanity.com/paper/2104.03204/) (2021)
- [A learned conditional prior for the VAE acoustic space of a TTS system](https://mlsanity.com/paper/2106.10229/) (2021)
- [SVSNet: An End-to-end Speaker Voice Similarity Assessment Model](https://mlsanity.com/paper/2107.09392/) (2021)
- [TGAVC: Improving Autoencoder Voice Conversion with Text-Guided and Adversarial Training](https://mlsanity.com/paper/2208.04035/) (2022)
- [Catch You and I Can: Revealing Source Voiceprint Against Voice Conversion](https://mlsanity.com/paper/2302.12434/) (2023)
- [Hear Your Face: Face-based voice conversion with F0 estimation](https://mlsanity.com/paper/2408.09802/) (2024)
- [Conditional Deep Hierarchical Variational Autoencoder for Voice Conversion](https://mlsanity.com/paper/2112.02796/) (2021)
- [Learning and controlling the source-filter representation of speech with a variational autoencoder](https://mlsanity.com/paper/2204.07075/) (2022)
- [Non-Parallel Voice Conversion for ASR Augmentation](https://mlsanity.com/paper/2209.06987/) (2022)
- [AlignSTS: Speech-to-Singing Conversion via Cross-Modal Alignment](https://mlsanity.com/paper/2305.04476/) (2023)
- [Zero-shot personalized lip-to-speech synthesis with face image based voice control](https://mlsanity.com/paper/2305.14359/) (2023)

## Abstract (arXiv)

We propose a flexible framework for spectral conversion (SC) that facilitates training with unaligned corpora. Many SC frameworks require parallel corpora, phonetic alignments, or explicit frame-wise correspondence for learning conversion functions or for synthesizing a target spectrum with the aid of alignments. However, these requirements gravely limit the scope of practical applications of SC due to scarcity or even unavailability of parallel corpora. We propose an SC framework based on variational auto-encoder which enables us to exploit non-parallel corpora. The framework comprises an encoder that learns speaker-independent phonetic representations and a decoder that learns to reconstruct the designated speaker. It removes the requirement of parallel corpora or phonetic alignments to train a spectral conversion system. We report objective and subjective evaluations to validate our proposed method and compare it to SC methods that have access to aligned corpora.