# Accent and Speaker Disentanglement in Many-to-many Voice Conversion

arXiv: https://arxiv.org/abs/2011.08609 · HTML: https://mlsanity.com/paper/2011.08609/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Nov 2020.

## Built on by (most cited first, 15 of 15)

- [Voice-preserving Zero-shot Multiple Accent Conversion](https://mlsanity.com/paper/2211.13282/) (2022)
- [Disentangleing Content and Fine-grained Prosody Information via Hybrid ASR Bottleneck Features for Voice Conversion](https://mlsanity.com/paper/2203.12813/) (2022)
- [Mix and Match: An Empirical Study on Training Corpus Composition for Polyglot Text-To-Speech (TTS)](https://mlsanity.com/paper/2207.01507/) (2022)
- [Modelling low-resource accents without accent-specific TTS frontend](https://mlsanity.com/paper/2301.04606/) (2023)
- [Disentangling segmental and prosodic factors to non-native speech comprehensibility](https://mlsanity.com/paper/2408.10997/) (2024)
- [DART: Disentanglement of Accent and Speaker Representation in Multispeaker Text-to-Speech](https://mlsanity.com/paper/2410.13342/) (2024)
- [REF-VC: Robust, Expressive and Fast Zero-Shot Voice Conversion with Diffusion Transformers](https://mlsanity.com/paper/2508.04996/) (2025)
- [Non-Parallel Voice Conversion for ASR Augmentation](https://mlsanity.com/paper/2209.06987/) (2022)
- [Accented Text-to-Speech Synthesis with a Conditional Variational Autoencoder](https://mlsanity.com/paper/2211.03316/) (2022)
- [LASPA: Language Agnostic Speaker Disentanglement with Prefix-Tuned Cross-Attention](https://mlsanity.com/paper/2506.02083/) (2025)
- [CorrectSpeech: A Fully Automated System for Speech Correction and Accent Reduction](https://mlsanity.com/paper/2204.05460/) (2022)
- [R2-SVC: Towards Real-World Robust and Expressive Zero-shot Singing Voice Conversion](https://mlsanity.com/paper/2510.20677/) (2025)
- [Cross-lingual Knowledge Distillation via Flow-based Voice Conversion for Robust Polyglot Text-To-Speech](https://mlsanity.com/paper/2309.08255/) (2023)
- [Accent Conversion in Text-To-Speech Using Multi-Level VAE and Adversarial Training](https://mlsanity.com/paper/2406.01018/) (2024)
- [Vibrato Expression Control for Singing Voice Conversion with Improving Independent Control](https://mlsanity.com/paper/2606.17126/) (2026)

## Abstract (arXiv)

This paper proposes an interesting voice and accent joint conversion approach, which can convert an arbitrary source speaker's voice to a target speaker with non-native accent. This problem is challenging as each target speaker only has training data in native accent and we need to disentangle accent and speaker information in the conversion model training and re-combine them in the conversion stage. In our recognition-synthesis conversion framework, we manage to solve this problem by two proposed tricks. First, we use accent-dependent speech recognizers to obtain bottleneck features for different accented speakers. This aims to wipe out other factors beyond the linguistic information in the BN features for conversion model training. Second, we propose to use adversarial training to better disentangle the speaker and accent information in our encoder-decoder based conversion model. Specifically, we plug an auxiliary speaker classifier to the encoder, trained with an adversarial loss to wipe out speaker information from the encoder output. Experiments show that our approach is superior to the baseline. The proposed tricks are quite effective in improving accentedness and audio quality and speaker similarity are well maintained.