# FreeVC: Towards High-Quality Text-Free One-Shot Voice Conversion

arXiv: https://arxiv.org/abs/2210.15418 · HTML: https://mlsanity.com/paper/2210.15418/

## Key facts

- Cited by 80 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 31 references that are in the map.
- First posted on arXiv 27 Oct 2022.

## Builds on (most cited first, 25 of 25)

- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [Least Squares Generative Adversarial Networks](https://mlsanity.com/paper/1611.04076/) (2016)
- [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://mlsanity.com/paper/1904.02882/) (2019)
- [Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech](https://mlsanity.com/paper/2106.06103/) (2021)
- [Unsupervised Cross-lingual Representation Learning for Speech Recognition](https://mlsanity.com/paper/2006.13979/) (2020)
- [Autoencoding beyond pixels using a learned similarity metric](https://mlsanity.com/paper/1512.09300/) (2015)
- [YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone](https://mlsanity.com/paper/2112.02418/) (2021)
- [Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis](https://mlsanity.com/paper/1803.09017/) (2018)
- [AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Loss](https://mlsanity.com/paper/1905.05879/) (2019)
- [Seen and Unseen emotional style transfer for voice conversion with a new emotional speech dataset](https://mlsanity.com/paper/2010.14794/) (2020)
- [Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised Representations](https://mlsanity.com/paper/2110.14513/) (2021)
- [Large-scale Self-Supervised Speech Representation Learning for Automatic Speaker Verification](https://mlsanity.com/paper/2110.05777/) (2021)
- [Voice Conversion Challenge 2020: Intra-lingual semi-parallel and cross-lingual voice conversion](https://mlsanity.com/paper/2008.12527/) (2020)
- [VQMIVC: Vector Quantization and Mutual Information-Based Unsupervised Speech Representation Disentanglement for One-shot Voice Conversion](https://mlsanity.com/paper/2106.10132/) (2021)
- [AGAIN-VC: A One-shot Voice Conversion using Activation Guidance and Adaptive Instance Normalization](https://mlsanity.com/paper/2011.00316/) (2020)
- [Any-to-Many Voice Conversion with Location-Relative Sequence-to-Sequence Modeling](https://mlsanity.com/paper/2009.02725/) (2020)
- [VQVC+: One-Shot Voice Conversion by Vector Quantization and U-Net architecture](https://mlsanity.com/paper/2006.04154/) (2020)
- [SpeechSplit 2.0: Unsupervised speech disentanglement for voice conversion Without tuning autoencoder Bottlenecks](https://mlsanity.com/paper/2203.14156/) (2022)
- [S3PRL-VC: Open-source Voice Conversion Framework with Self-supervised Speech Representations](https://mlsanity.com/paper/2110.06280/) (2021)
- [Transfer Learning from Speech Synthesis to Voice Conversion with Non-Parallel Training Data](https://mlsanity.com/paper/2009.14399/) (2020)
- [Cotatron: Transcription-Guided Speech Encoder for Any-to-Many Voice Conversion without Parallel Data](https://mlsanity.com/paper/2005.03295/) (2020)
- [Collapsed speech segment detection and suppression for WaveNet vocoder](https://mlsanity.com/paper/1804.11055/) (2018)
- [GlowVC: Mel-spectrogram space disentangling model for language-independent text-free voice conversion](https://mlsanity.com/paper/2207.01454/) (2022)

## Built on by (most cited first, 30 of 80)

- [MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformer](https://mlsanity.com/paper/2409.00750/) (2024)
- [Voice Conversion With Just Nearest Neighbors](https://mlsanity.com/paper/2305.18975/) (2023)
- [Zero-shot Voice Conversion with Diffusion Transformers](https://mlsanity.com/paper/2411.09943/) (2024)
- [OpenVoice: Versatile Instant Voice Cloning](https://mlsanity.com/paper/2312.01479/) (2023)
- [Coding Speech through Vocal Tract Kinematics](https://mlsanity.com/paper/2406.12998/) (2024)
- [Susceptibility to Influence of Large Language Models](https://mlsanity.com/paper/2303.06074/) (2023)
- [PolyGlotFake: A Novel Multilingual and Multimodal DeepFake Dataset](https://mlsanity.com/paper/2405.08838/) (2024)
- [StreamVC: Real-Time Low-Latency Voice Conversion](https://mlsanity.com/paper/2401.03078/) (2024)
- [Fake the Real: Backdoor Attack on Deep Speech Classification via Voice Conversion](https://mlsanity.com/paper/2306.15875/) (2023)
- [Predicting Turn-Taking and Backchannel in Human-Machine Conversations Using Linguistic, Acoustic, and Visual Signals](https://mlsanity.com/paper/2505.12654/) (2025)
- [Phoneme Hallucinator: One-shot Voice Conversion via Set Expansion](https://mlsanity.com/paper/2308.06382/) (2023)
- [Voice Attribute Editing with Text Prompt](https://mlsanity.com/paper/2404.08857/) (2024)
- [CLAD: Robust Audio Deepfake Detection Against Manipulation Attacks with Contrastive Learning](https://mlsanity.com/paper/2404.15854/) (2024)
- [Enhancing Partially Spoofed Audio Localization with Boundary-aware Attention Mechanism](https://mlsanity.com/paper/2407.21611/) (2024)
- [Fake Artificial Intelligence Generated Contents (FAIGC): A Survey of Theories, Detection Methods, and Opportunities](https://mlsanity.com/paper/2405.00711/) (2024)
- [GenVC: Self-Supervised Zero-Shot Voice Conversion](https://mlsanity.com/paper/2502.04519/) (2025)
- [The Third VoicePrivacy Challenge: Preserving Emotional Expressiveness and Linguistic Content in Voice Anonymization](https://mlsanity.com/paper/2601.11846/) (2026)
- [Enhancing the Stability of LLM-based Speech Generation Systems through Self-Supervised Representations](https://mlsanity.com/paper/2402.03407/) (2024)
- [VoiceShop: A Unified Speech-to-Speech Framework for Identity-Preserving Zero-Shot Voice Editing](https://mlsanity.com/paper/2404.06674/) (2024)
- [VQ-CTAP: Cross-Modal Fine-Grained Sequence Representation Learning for Speech Processing](https://mlsanity.com/paper/2408.05758/) (2024)
- [vec2wav 2.0: Advancing Voice Conversion via Discrete Token Vocoders](https://mlsanity.com/paper/2409.01995/) (2024)
- [SaMoye: Zero-shot Singing Voice Conversion Model Based on Feature Disentanglement and Enhancement](https://mlsanity.com/paper/2407.07728/) (2024)
- [Improving Pronunciation and Accent Conversion through Knowledge Distillation And Synthetic Ground-Truth from Native TTS](https://mlsanity.com/paper/2410.14997/) (2024)
- [An Exhaustive Evaluation of TTS- and VC-based Data Augmentation for ASR](https://mlsanity.com/paper/2503.08954/) (2025)
- [SpecWav-Attack: Leveraging Spectrogram Resizing and Wav2Vec 2.0 for Attacking Anonymized Speech](https://mlsanity.com/paper/2505.09616/) (2025)
- [TVTSyn: Content-Synchronous Time-Varying Timbre for Streaming Voice Conversion and Anonymization](https://mlsanity.com/paper/2602.09389/) (2026)
- [Voice Conversion for Stuttered Speech, Instruments, Unseen Languages and Textually Described Voices](https://mlsanity.com/paper/2310.08104/) (2023)
- [Vulnerability of Automatic Identity Recognition to Audio-Visual Deepfakes](https://mlsanity.com/paper/2311.17655/) (2023)
- [SF-Speech: Straightened Flow for Zero-Shot Voice Clone](https://mlsanity.com/paper/2410.12399/) (2024)
- [Optimal Transport Maps are Good Voice Converters](https://mlsanity.com/paper/2411.02402/) (2024)

## Abstract (arXiv)

Voice conversion (VC) can be achieved by first extracting source content information and target speaker information, and then reconstructing waveform with these information. However, current approaches normally either extract dirty content information with speaker information leaked in, or demand a large amount of annotated data for training. Besides, the quality of reconstructed waveform can be degraded by the mismatch between conversion model and vocoder. In this paper, we adopt the end-to-end framework of VITS for high-quality waveform reconstruction, and propose strategies for clean content information extraction without text annotation. We disentangle content information by imposing an information bottleneck to WavLM features, and propose the spectrogram-resize based data augmentation to improve the purity of extracted content information. Experimental results show that the proposed method outperforms the latest VC models trained with annotated data and has greater robustness.