# SpeechSplit 2.0: Unsupervised speech disentanglement for voice conversion Without tuning autoencoder Bottlenecks

arXiv: https://arxiv.org/abs/2203.14156 · HTML: https://mlsanity.com/paper/2203.14156/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 20 references that are in the map.
- First posted on arXiv 26 Mar 2022.

## Builds on (most cited first, 12 of 12)

- [StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation](https://mlsanity.com/paper/1711.09020/) (2017)
- [AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Loss](https://mlsanity.com/paper/1905.05879/) (2019)
- [Unsupervised Speech Decomposition via Triple Information Bottleneck](https://mlsanity.com/paper/2004.11284/) (2020)
- [Voice Conversion from Non-parallel Corpora Using Variational Auto-encoder](https://mlsanity.com/paper/1610.04019/) (2016)
- [StarGAN-VC2: Rethinking Conditional Methods for StarGAN-Based Voice Conversion](https://mlsanity.com/paper/1907.12279/) (2019)
- [F0-consistent many-to-many non-parallel voice conversion via conditional autoencoder](https://mlsanity.com/paper/2004.07370/) (2020)
- [Investigation of Data Augmentation Techniques for Disordered Speech Recognition](https://mlsanity.com/paper/2201.05562/) (2022)
- [Multi-target Voice Conversion without Parallel Data by Adversarially Learning Disentangled Audio Representations](https://mlsanity.com/paper/1804.02812/) (2018)
- [VAW-GAN for Disentanglement and Recomposition of Emotional Elements in Speech](https://mlsanity.com/paper/2011.02314/) (2020)
- [Can Speaker Augmentation Improve Multi-Speaker End-to-End TTS?](https://mlsanity.com/paper/2005.01245/) (2020)
- [Unsupervised Adaptation with Interpretable Disentangled Representations for Distant Conversational Speech Recognition](https://mlsanity.com/paper/1806.04872/) (2018)
- [Protecting gender and identity with disentangled speech representations](https://mlsanity.com/paper/2104.11051/) (2021)

## Built on by (most cited first, 28 of 28)

- [FreeVC: Towards High-Quality Text-Free One-Shot Voice Conversion](https://mlsanity.com/paper/2210.15418/) (2022)
- [SEF-VC: Speaker Embedding Free Zero-Shot Voice Conversion with Cross Attention](https://mlsanity.com/paper/2312.08676/) (2023)
- [Speech Representation Disentanglement with Adversarial Mutual Information Learning for One-shot Voice Conversion](https://mlsanity.com/paper/2208.08757/) (2022)
- [An Efficient Temporary Deepfake Location Approach Based Embeddings for Partially Spoofed Audio Detection](https://mlsanity.com/paper/2309.03036/) (2023)
- [End-to-end streaming model for low-latency speech anonymization](https://mlsanity.com/paper/2406.09277/) (2024)
- [Disentangling Prosody Representations with Unsupervised Speech Reconstruction](https://mlsanity.com/paper/2212.06972/) (2022)
- [LSCodec: Low-Bitrate and Speaker-Decoupled Discrete Speech Codec](https://mlsanity.com/paper/2410.15764/) (2024)
- [vec2wav 2.0: Advancing Voice Conversion via Discrete Token Vocoders](https://mlsanity.com/paper/2409.01995/) (2024)
- [Generalized Source Tracing: Detecting Novel Audio Deepfake Algorithm with Real Emphasis and Fake Dispersion Strategy](https://mlsanity.com/paper/2406.03240/) (2024)
- [Imperceptible Rhythm Backdoor Attacks: Exploring Rhythm Transformation for Embedding Undetectable Vulnerabilities on Speech Recognition](https://mlsanity.com/paper/2406.10932/) (2024)
- [Discl-VC: Disentangled Discrete Tokens and In-Context Learning for Controllable Zero-Shot Voice Conversion](https://mlsanity.com/paper/2505.24291/) (2025)
- [TVTSyn: Content-Synchronous Time-Varying Timbre for Streaming Voice Conversion and Anonymization](https://mlsanity.com/paper/2602.09389/) (2026)
- [A unified one-shot prosody and speaker conversion system with self-supervised discrete speech units](https://mlsanity.com/paper/2211.06535/) (2022)
- [PSCodec: A Series of High-Fidelity Low-bitrate Neural Speech Codecs Leveraging Prompt Encoders](https://mlsanity.com/paper/2404.02702/) (2024)
- [Representation Purification for End-to-End Speech Translation](https://mlsanity.com/paper/2412.04266/) (2024)
- [ZSVC: Zero-shot Style Voice Conversion with Disentangled Latent Diffusion Models and Adversarial Training](https://mlsanity.com/paper/2501.04416/) (2025)
- [Improved disentangled speech representations using contrastive learning in factorized hierarchical variational autoencoder](https://mlsanity.com/paper/2211.08191/) (2022)
- [AlignSTS: Speech-to-Singing Conversion via Cross-Modal Alignment](https://mlsanity.com/paper/2305.04476/) (2023)
- [Towards Evaluating the Robustness of Automatic Speech Recognition Systems via Audio Style Transfer](https://mlsanity.com/paper/2405.09470/) (2024)
- [Whispering Under the Eaves: Protecting User Privacy Against Commercial and LLM-powered Automatic Speech Recognition Systems](https://mlsanity.com/paper/2504.00858/) (2025)
- [Voice Conversion for Lombard Speaking Style with Implicit and Explicit Acoustic Feature Conditioning](https://mlsanity.com/paper/2507.09310/) (2025)
- [NE-PADD: Leveraging Named Entity Knowledge for Robust Partial Audio Deepfake Detection via Attention Aggregation](https://mlsanity.com/paper/2509.03829/) (2025)
- [Automatic Speech Disentanglement for Voice Conversion using Rank Module and Speech Augmentation](https://mlsanity.com/paper/2306.12259/) (2023)
- [An Unsupervised Domain Adaptation Method for Locating Manipulated Region in partially fake Audio](https://mlsanity.com/paper/2407.08239/) (2024)
- [Pureformer-VC: Non-parallel One-Shot Voice Conversion with Pure Transformer Blocks and Triplet Discriminative Training](https://mlsanity.com/paper/2409.01668/) (2024)
- [StyleStream: Real-Time Zero-Shot Voice Style Conversion](https://mlsanity.com/paper/2602.20113/) (2026)
- [Do speech foundation models really learn words?](https://mlsanity.com/paper/2609.10434/) (2026)
- [EMODE: Dynamic Para-Semantic Experts for Emotion-Aware Speech Language Modeling](https://mlsanity.com/paper/2610.06956/) (2026)

## Abstract (arXiv)

SpeechSplit can perform aspect-specific voice conversion by disentangling speech into content, rhythm, pitch, and timbre using multiple autoencoders in an unsupervised manner. However, SpeechSplit requires careful tuning of the autoencoder bottlenecks, which can be time-consuming and less robust. This paper proposes SpeechSplit 2.0, which constrains the information flow of the speech component to be disentangled on the autoencoder input using efficient signal processing methods instead of bottleneck tuning. Evaluation results show that SpeechSplit 2.0 achieves comparable performance to SpeechSplit in speech disentanglement and superior robustness to the bottleneck size variations. Our code is available at https://github.com/biggytruck/SpeechSplit2.