# VQVC+: One-Shot Voice Conversion by Vector Quantization and U-Net architecture

arXiv: https://arxiv.org/abs/2006.04154 · HTML: https://mlsanity.com/paper/2006.04154/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Jun 2020.

## Built on by (most cited first, 30 of 35)

- [Diffusion-Based Voice Conversion with Fast Maximum Likelihood Sampling Scheme](https://mlsanity.com/paper/2109.13821/) (2021)
- [Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised Representations](https://mlsanity.com/paper/2110.14513/) (2021)
- [FreeVC: Towards High-Quality Text-Free One-Shot Voice Conversion](https://mlsanity.com/paper/2210.15418/) (2022)
- [VQMIVC: Vector Quantization and Mutual Information-Based Unsupervised Speech Representation Disentanglement for One-shot Voice Conversion](https://mlsanity.com/paper/2106.10132/) (2021)
- [Vision+X: A Survey on Multimodal Learning in the Light of Data](https://mlsanity.com/paper/2210.02884/) (2022)
- [Speech Representation Disentanglement with Adversarial Mutual Information Learning for One-shot Voice Conversion](https://mlsanity.com/paper/2208.08757/) (2022)
- ["Hello, It's Me": Deep Learning-based Speech Synthesis Attacks in the Real World](https://mlsanity.com/paper/2109.09598/) (2021)
- [Voice Filter: Few-shot text-to-speech speaker adaptation using voice conversion as a post-processing module](https://mlsanity.com/paper/2202.08164/) (2022)
- [Many-to-Many Voice Conversion based Feature Disentanglement using Variational Autoencoder](https://mlsanity.com/paper/2107.06642/) (2021)
- [VQ-CTAP: Cross-Modal Fine-Grained Sequence Representation Learning for Speech Processing](https://mlsanity.com/paper/2408.05758/) (2024)
- [Zero-shot Voice Conversion via Self-supervised Prosody Representation Learning](https://mlsanity.com/paper/2110.14422/) (2021)
- [Retriever: Learning Content-Style Representation as a Token-Level Bipartite Graph](https://mlsanity.com/paper/2202.12307/) (2022)
- [Robust One-Shot Singing Voice Conversion](https://mlsanity.com/paper/2210.11096/) (2022)
- [CSLP-AE: A Contrastive Split-Latent Permutation Autoencoder Framework for Zero-Shot Electroencephalography Signal Conversion](https://mlsanity.com/paper/2311.07788/) (2023)
- [SIG-VC: A Speaker Information Guided Zero-shot Voice Conversion System for Both Human Beings and Machines](https://mlsanity.com/paper/2111.03811/) (2021)
- [NoiseVC: Towards High Quality Zero-Shot Voice Conversion](https://mlsanity.com/paper/2104.06074/) (2021)
- [TGAVC: Improving Autoencoder Voice Conversion with Text-Guided and Adversarial Training](https://mlsanity.com/paper/2208.04035/) (2022)
- [Catch You and I Can: Revealing Source Voiceprint Against Voice Conversion](https://mlsanity.com/paper/2302.12434/) (2023)
- [DualVC: Dual-mode Voice Conversion using Intra-model Knowledge Distillation and Hybrid Predictive Coding](https://mlsanity.com/paper/2305.12425/) (2023)
- [SecureSpectra: Safeguarding Digital Identity from Deep Fake Threats via Intelligent Signatures](https://mlsanity.com/paper/2407.00913/) (2024)
- [A unified one-shot prosody and speaker conversion system with self-supervised discrete speech units](https://mlsanity.com/paper/2211.06535/) (2022)
- [Reimagining Speech: A Scoping Review of Deep Learning-Powered Voice Conversion](https://mlsanity.com/paper/2311.08104/) (2023)
- [Disentangling segmental and prosodic factors to non-native speech comprehensibility](https://mlsanity.com/paper/2408.10997/) (2024)
- [OCTOPUS: Overcoming Performance andPrivatization Bottlenecks in Distributed Learning](https://mlsanity.com/paper/2105.00602/) (2021)
- [Conditional Deep Hierarchical Variational Autoencoder for Voice Conversion](https://mlsanity.com/paper/2112.02796/) (2021)
- [Iteratively Improving Speech Recognition and Voice Conversion](https://mlsanity.com/paper/2305.15055/) (2023)
- [Cross-modal Face- and Voice-style Transfer](https://mlsanity.com/paper/2302.13838/) (2023)
- [Discrete Unit based Masking for Improving Disentanglement in Voice Conversion](https://mlsanity.com/paper/2409.11560/) (2024)
- [Takin-VC: Expressive Zero-Shot Voice Conversion via Adaptive Hybrid Content Encoding and Enhanced Timbre Modeling](https://mlsanity.com/paper/2410.01350/) (2024)
- [Rhythm Controllable and Efficient Zero-Shot Voice Conversion via Shortcut Flow Matching](https://mlsanity.com/paper/2506.01014/) (2025)

## Abstract (arXiv)

Voice conversion (VC) is a task that transforms the source speaker's timbre, accent, and tones in audio into another one's while preserving the linguistic content. It is still a challenging work, especially in a one-shot setting. Auto-encoder-based VC methods disentangle the speaker and the content in input speech without given the speaker's identity, so these methods can further generalize to unseen speakers. The disentangle capability is achieved by vector quantization (VQ), adversarial training, or instance normalization (IN). However, the imperfect disentanglement may harm the quality of output speech. In this work, to further improve audio quality, we use the U-Net architecture within an auto-encoder-based VC system. We find that to leverage the U-Net architecture, a strong information bottleneck is necessary. The VQ-based method, which quantizes the latent vectors, can serve the purpose. The objective and the subjective evaluations show that the proposed method performs well in both audio naturalness and speaker similarity.