# Adversarial Speaker Disentanglement Using Unannotated External Data for Self-supervised Representation Based Voice Conversion

arXiv: https://arxiv.org/abs/2305.09167 · HTML: https://mlsanity.com/paper/2305.09167/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 20 references that are in the map.
- First posted on arXiv 16 May 2023.

## Builds on (most cited first, 17 of 17)

- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Loss](https://mlsanity.com/paper/1905.05879/) (2019)
- [AISHELL-3: A Multi-speaker Mandarin TTS Corpus and the Baselines](https://mlsanity.com/paper/2010.11567/) (2020)
- [Wespeaker: A Research and Production oriented Speaker Embedding Learning Toolkit](https://mlsanity.com/paper/2210.17016/) (2022)
- [One-shot Voice Conversion by Separating Speaker and Content Representations with Instance Normalization](https://mlsanity.com/paper/1904.05742/) (2019)
- [Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised Representations](https://mlsanity.com/paper/2110.14513/) (2021)
- [ContentVec: An Improved Self-Supervised Speech Representation by Disentangling Speakers](https://mlsanity.com/paper/2204.09224/) (2022)
- [A Comparison of Discrete and Soft Speech Units for Improved Voice Conversion](https://mlsanity.com/paper/2111.02392/) (2021)
- [Analyzing Speaker Information in Self-Supervised Models to Improve Zero-Resource Speech Processing](https://mlsanity.com/paper/2108.00917/) (2021)
- [FastSVC: Fast Cross-Domain Singing Voice Conversion with Feature-wise Linear Modulation](https://mlsanity.com/paper/2011.05731/) (2020)
- [Adversarially learning disentangled speech representations for robust multi-factor voice conversion](https://mlsanity.com/paper/2102.00184/) (2021)
- [Speaker Normalization for Self-supervised Speech Emotion Recognition](https://mlsanity.com/paper/2202.01252/) (2022)
- [Disentangleing Content and Fine-grained Prosody Information via Hybrid ASR Bottleneck Features for Voice Conversion](https://mlsanity.com/paper/2203.12813/) (2022)
- [Towards High-fidelity Singing Voice Conversion with Acoustic Reference and Contrastive Predictive Coding](https://mlsanity.com/paper/2110.04754/) (2021)
- [One-shot Voice Conversion For Style Transfer Based On Speaker Adaptation](https://mlsanity.com/paper/2111.12277/) (2021)

## Built on by (most cited first, 1 of 1)

- [Singing Voice Conversion with Accompaniment Using Self-Supervised Representation-Based Melody Features](https://mlsanity.com/paper/2502.04722/) (2025)

## Abstract (arXiv)

Nowadays, recognition-synthesis-based methods have been quite popular with voice conversion (VC). By introducing linguistics features with good disentangling characters extracted from an automatic speech recognition (ASR) model, the VC performance achieved considerable breakthroughs. Recently, self-supervised learning (SSL) methods trained with a large-scale unannotated speech corpus have been applied to downstream tasks focusing on the content information, which is suitable for VC tasks. However, a huge amount of speaker information in SSL representations degrades timbre similarity and the quality of converted speech significantly. To address this problem, we proposed a high-similarity any-to-one voice conversion method with the input of SSL representations. We incorporated adversarial training mechanisms in the synthesis module using external unannotated corpora. Two auxiliary discriminators were trained to distinguish whether a sequence of mel-spectrograms has been converted by the acoustic model and whether a sequence of content embeddings contains speaker information from external corpora. Experimental results show that our proposed method achieves comparable similarity and higher naturalness than the supervised method, which needs a huge amount of annotated corpora for training and is applicable to improve similarity for VC methods with other SSL representations as input.