# S2VC: A Framework for Any-to-Any Voice Conversion with Self-Supervised Pretrained Representations

arXiv: https://arxiv.org/abs/2104.02901 · HTML: https://mlsanity.com/paper/2104.02901/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Apr 2021.

## Built on by (most cited first, 17 of 17)

- [S3PRL-VC: Open-source Voice Conversion Framework with Self-supervised Speech Representations](https://mlsanity.com/paper/2110.06280/) (2021)
- [SpeechPrompt: An Exploration of Prompt Tuning on Generative Spoken Language Model for Speech Processing Tasks](https://mlsanity.com/paper/2203.16773/) (2022)
- [SEF-VC: Speaker Embedding Free Zero-Shot Voice Conversion with Cross Attention](https://mlsanity.com/paper/2312.08676/) (2023)
- [Speaking Style Conversion in the Waveform Domain Using Discrete Self-Supervised Units](https://mlsanity.com/paper/2212.09730/) (2022)
- [A Comparative Study of Self-supervised Speech Representation Based Voice Conversion](https://mlsanity.com/paper/2207.04356/) (2022)
- [GenVC: Self-Supervised Zero-Shot Voice Conversion](https://mlsanity.com/paper/2502.04519/) (2025)
- [Training Robust Zero-Shot Voice Conversion Models with Self-supervised Features](https://mlsanity.com/paper/2112.04424/) (2021)
- [FMFCC-A: A Challenging Mandarin Dataset for Synthetic Speech Detection](https://mlsanity.com/paper/2110.09441/) (2021)
- [Retriever: Learning Content-Style Representation as a Token-Level Bipartite Graph](https://mlsanity.com/paper/2202.12307/) (2022)
- [Towards General-Purpose Text-Instruction-Guided Voice Conversion](https://mlsanity.com/paper/2309.14324/) (2023)
- [Toward Degradation-Robust Voice Conversion](https://mlsanity.com/paper/2110.07537/) (2021)
- [ELP-Adapters: Parameter Efficient Adapter Tuning for Various Speech Processing Tasks](https://mlsanity.com/paper/2407.21066/) (2024)
- [REF-VC: Robust, Expressive and Fast Zero-Shot Voice Conversion with Diffusion Transformers](https://mlsanity.com/paper/2508.04996/) (2025)
- [Emotion Recognition and Generation: A Comprehensive Review of Face, Speech, and Text Modalities](https://mlsanity.com/paper/2502.06803/) (2025)
- [Converting Anyone's Voice: End-to-End Expressive Voice Conversion with a Conditional Diffusion Model](https://mlsanity.com/paper/2405.01730/) (2024)
- [Takin-VC: Expressive Zero-Shot Voice Conversion via Adaptive Hybrid Content Encoding and Enhanced Timbre Modeling](https://mlsanity.com/paper/2410.01350/) (2024)
- [CycleFlow: Purify Information Factors by Cycle Loss](https://mlsanity.com/paper/2110.09928/) (2021)

## Abstract (arXiv)

Any-to-any voice conversion (VC) aims to convert the timbre of utterances from and to any speakers seen or unseen during training. Various any-to-any VC approaches have been proposed like AUTOVC, AdaINVC, and FragmentVC. AUTOVC, and AdaINVC utilize source and target encoders to disentangle the content and speaker information of the features. FragmentVC utilizes two encoders to encode source and target information and adopts cross attention to align the source and target features with similar phonetic content. Moreover, pre-trained features are adopted. AUTOVC used dvector to extract speaker information, and self-supervised learning (SSL) features like wav2vec 2.0 is used in FragmentVC to extract the phonetic content information. Different from previous works, we proposed S2VC that utilizes Self-Supervised features as both source and target features for VC model. Supervised phoneme posteriororgram (PPG), which is believed to be speaker-independent and widely used in VC to extract content information, is chosen as a strong baseline for SSL features. The objective evaluation and subjective evaluation both show models taking SSL feature CPC as both source and target features outperforms that taking PPG as source feature, suggesting that SSL features have great potential in improving VC.