# Unsupervised Speech Decomposition via Triple Information Bottleneck

arXiv: https://arxiv.org/abs/2004.11284 · HTML: https://mlsanity.com/paper/2004.11284/

## Key facts

- Cited by 88 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Apr 2020.

## Built on by (most cited first, 30 of 88)

- [NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models](https://mlsanity.com/paper/2403.03100/) (2024)
- [A Survey on Neural Speech Synthesis](https://mlsanity.com/paper/2106.15561/) (2021)
- [SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Models](https://mlsanity.com/paper/2308.16692/) (2023)
- [ContentVec: An Improved Self-Supervised Speech Representation by Disentangling Speakers](https://mlsanity.com/paper/2204.09224/) (2022)
- [VQMIVC: Vector Quantization and Mutual Information-Based Unsupervised Speech Representation Disentanglement for One-shot Voice Conversion](https://mlsanity.com/paper/2106.10132/) (2021)
- [PromptTTS 2: Describing and Generating Voices with Text Prompt](https://mlsanity.com/paper/2309.02285/) (2023)
- [Improving Subgraph Recognition with Variational Graph Information Bottleneck](https://mlsanity.com/paper/2112.09899/) (2021)
- [DDDM-VC: Decoupled Denoising Diffusion Models with Disentangled Representation and Prior Mixup for Verified Robust Voice Conversion](https://mlsanity.com/paper/2305.15816/) (2023)
- [SpeechGPT-Gen: Scaling Chain-of-Information Speech Generation](https://mlsanity.com/paper/2401.13527/) (2024)
- [Live Speech Portraits: Real-Time Photorealistic Talking-Head Animation](https://mlsanity.com/paper/2109.10595/) (2021)
- [TranSpeech: Speech-to-Speech Translation With Bilateral Perturbation](https://mlsanity.com/paper/2205.12523/) (2022)
- [Diff-HierVC: Diffusion-based Hierarchical Voice Conversion with Robust Pitch Generation and Masked Prior for Zero-shot Speaker Adaptation](https://mlsanity.com/paper/2311.04693/) (2023)
- [Textless Speech Emotion Conversion using Discrete and Decomposed Representations](https://mlsanity.com/paper/2111.07402/) (2021)
- [Recognizing Predictive Substructures with Subgraph Information Bottleneck](https://mlsanity.com/paper/2103.11155/) (2021)
- [SpeechSplit 2.0: Unsupervised speech disentanglement for voice conversion Without tuning autoencoder Bottlenecks](https://mlsanity.com/paper/2203.14156/) (2022)
- [Make-A-Voice: Unified Voice Synthesis With Discrete Representation](https://mlsanity.com/paper/2305.19269/) (2023)
- [GenerSpeech: Towards Style Transfer for Generalizable Out-Of-Domain Text-to-Speech](https://mlsanity.com/paper/2205.07211/) (2022)
- [Multimodal Emotion Recognition with High-level Speech and Text Features](https://mlsanity.com/paper/2111.10202/) (2021)
- [SEF-VC: Speaker Embedding Free Zero-Shot Voice Conversion with Cross Attention](https://mlsanity.com/paper/2312.08676/) (2023)
- [STYLER: Style Factor Modeling with Rapidity and Robustness via Speech Decomposition for Expressive and Controllable Neural Text to Speech](https://mlsanity.com/paper/2103.09474/) (2021)
- [Shepherding Slots to Objects: Towards Stable and Robust Object-Centric Learning](https://mlsanity.com/paper/2303.17842/) (2023)
- [Speaking Style Conversion in the Waveform Domain Using Discrete Self-Supervised Units](https://mlsanity.com/paper/2212.09730/) (2022)
- [Adversarially learning disentangled speech representations for robust multi-factor voice conversion](https://mlsanity.com/paper/2102.00184/) (2021)
- [Speech Representation Disentanglement with Adversarial Mutual Information Learning for One-shot Voice Conversion](https://mlsanity.com/paper/2208.08757/) (2022)
- [Self-Supervised Visual Acoustic Matching](https://mlsanity.com/paper/2307.15064/) (2023)
- ["Hello, It's Me": Deep Learning-based Speech Synthesis Attacks in the Real World](https://mlsanity.com/paper/2109.09598/) (2021)
- [Assem-VC: Realistic Voice Conversion by Assembling Modern Speech Synthesis Techniques](https://mlsanity.com/paper/2104.00931/) (2021)
- [Style Equalization: Unsupervised Learning of Controllable Generative Sequence Models](https://mlsanity.com/paper/2110.02891/) (2021)
- [VarietySound: Timbre-Controllable Video to Sound Generation via Unsupervised Information Disentanglement](https://mlsanity.com/paper/2211.10666/) (2022)
- [Disentangleing Content and Fine-grained Prosody Information via Hybrid ASR Bottleneck Features for Voice Conversion](https://mlsanity.com/paper/2203.12813/) (2022)

## Abstract (arXiv)

Speech information can be roughly decomposed into four components: language content, timbre, pitch, and rhythm. Obtaining disentangled representations of these components is useful in many speech analysis and generation applications. Recently, state-of-the-art voice conversion systems have led to speech representations that can disentangle speaker-dependent and independent information. However, these systems can only disentangle timbre, while information about pitch, rhythm and content is still mixed together. Further disentangling the remaining speech components is an under-determined problem in the absence of explicit annotations for each component, which are difficult and expensive to obtain. In this paper, we propose SpeechSplit, which can blindly decompose speech into its four components by introducing three carefully designed information bottlenecks. SpeechSplit is among the first algorithms that can separately perform style transfer on timbre, pitch and rhythm without text labels. Our code is publicly available at https://github.com/auspicious3000/SpeechSplit.