# Learning Expressive Disentangled Speech Representations with Soft Speech Units and Adversarial Style Augmentation

arXiv: https://arxiv.org/abs/2405.00603 · HTML: https://mlsanity.com/paper/2405.00603/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 May 2024.

## Built on by (most cited first, 1 of 1)

- [Fast-VGAN: Lightweight Voice Conversion with Explicit Control of F0 and Duration Parameters](https://mlsanity.com/paper/2507.04817/) (2025)

## Abstract (arXiv)

Voice conversion is the task to transform voice characteristics of source speech while preserving content information. Nowadays, self-supervised representation learning models are increasingly utilized in content extraction. However, in these representations, a lot of hidden speaker information leads to timbre leakage while the prosodic information of hidden units lacks use. To address these issues, we propose a novel framework for expressive voice conversion called "SAVC" based on soft speech units from HuBert-soft. Taking soft speech units as input, we design an attribute encoder to extract content and prosody features respectively. Specifically, we first introduce statistic perturbation imposed by adversarial style augmentation to eliminate speaker information. Then the prosody is implicitly modeled on soft speech units with knowledge distillation. Experiment results show that the intelligibility and naturalness of converted speech outperform previous work.